Working on way to subset fonts for ePub/KF3

[Freeshadow]

Originally Posted by Man Eating Duck

(I already have a script somewhere which creates a list of used characters)

For reasons presented in this thread here:
http://www.mobileread.com/forums/sho...49#post2240749

the legal and practical need of such a script is currently discussed ibid.

[JSWolf]

Originally Posted by Man Eating Duck

(I already have a script somewhere which creates a list of used characters)

What language does this script use? It is something that can easily by used with Windows 7? If so, can you please post it?

[Man Eating Duck]

Originally Posted by JSWolf

What language does this script use? It is something that can easily by used with Windows 7? If so, can you please post it?

I believe I wrote it in php, which can be installed on Win7. It just accepts an utf-8 input text file and spits out a character list to stdout, it's not very sophisticated. I'll see if I find it when I go to work tomorrow, if not I can probably rewrite it in a few minutes if you're still interested

[Man Eating Duck]

@JSWolf: Seems it was Python, it works in Python 2.7. I've probably done horrible things to the Python language, but here it is, no guarantuees about anything:

import argparse, codecs
parser = argparse.ArgumentParser(description='''This script will accept utf-8 text files and write a list of unique characters to stdout or an output file''')
parser.add_argument("file", nargs='+',help="input (utf-8) file(s) for character counting")
parser.add_argument("-o", "--outfile", help="outputfile")
args = parser.parse_args()
disallowed = set('')
s=set()
for f in args.file:
	s=s|set(char for line in codecs.open(f, encoding="UTF-8") for char in line 
         if char not in disallowed)
if args.outfile:
	print 'Writing to file: '+args.outfile;
	with codecs.open(args.outfile, "w", "utf-8") as f:
		f.write(u''.join(s))
		f.close
else:
	print u''.join(s).encode('utf-8')

usage: uniquechars.py [-h] [-o OUTFILE] file [file ...]

Use the output file option for Unicode files, as many glyhs won't show in a console.

If you have questions about the code I can try to answer, but I heard there are some guys in the calibre forum who probably have a bit more experience with Python

[Freeshadow]

Python means fontforge could be fed with it just what I tought about.

[DiapDealer]

It'd be nice to eliminate all characters from the script that occur inside html tags. Those wouldn't necessarily need to be a part of any embedded font since they won't be rendered.

[Man Eating Duck]

Originally Posted by DiapDealer

It'd be nice to eliminate all characters from the script that occur inside html tags. Those wouldn't necessarily need to be a part of any embedded font since they won't be rendered.

My (rather stupid) script expects pure utf-8 text files. You could get those by converting an epub to txt in calibre (remember to specify utf-8 as output encoding). Most authoring software can probably save to txt as well. Formatting doesn't really matter as long as every character is included. This could maybe have been more convenient, but parsing html is outside of my abilities, and I want those results before making an epub as well.

Since you might be interested only in special characters, you could just add a bunch of regular characters that you're not interested in to disallowed = set('') in line 6, ie

disallowed = set('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789')

This should exclude them from the results. Add as many as you feel like.

The script wasn't really intended for publication, so it's unfortunately pretty rough, and I don't really have enough experience to improve it. It works for my needs, though

[DiapDealer]

Spoiler:

#!/usr/bin/env python
# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai

import sys, argparse, codecs
from HTMLParser import HTMLParser
from re import sub
from sys import stderr
from traceback import print_exc

class _DeHTMLParser(HTMLParser):
	def __init__(self):
		HTMLParser.__init__(self)
		self.__text = []

	def handle_data(self, data):
		text = data.strip()
		if len(text) > 0:
			text = sub('[ \t\r\n]+', ' ', text)
			self.__text.append(text + ' ')

	def handle_starttag(self, tag, attrs):
		if tag == 'p':
			self.__text.append('\n\n')
		elif tag == 'br':
			self.__text.append('\n')

	def handle_startendtag(self, tag, attrs):
		if tag == 'br':
			self.__text.append('\n\n')

	def text(self):
		return ''.join(self.__text).strip()

def main():
	parser = argparse.ArgumentParser(description='''This script will accept utf-8 text files and write a list of unique characters to stdout or an output file''')
	parser.add_argument("file", nargs='+',help="input (utf-8) file(s) for character counting")
	parser.add_argument("-o", "--outfile", help="outputfile")
	parser.add_argument("-c", "--codec", help="input char encoding")
	args = parser.parse_args()
	disallowed = set('')
	s=set()
	if not args.codec:
		file_codec = 'utf-8'
	else:
		file_codec = args.codec
	for f in args.file:
		try:
			html_parser = _DeHTMLParser()
			html_parser.feed(unicode(file(f, 'r').read(), file_codec))
			html_parser.close()
			text = html_parser.text()
			s=s|set(char for line in text for char in line 
					if char not in disallowed)
		except:
			print_exc(file=stderr)
	if args.outfile:
		print 'Writing to file: '+args.outfile;
		with codecs.open(args.outfile, 'w', file_codec) as f:
			f.write(''.join(s))
			f.close
	else:
		print ''.join(s).encode(file_codec)
	
if __name__ == '__main__':
	sys.exit(main())

usage: uniquechars.py [-h] [-c CODEC] [-o OUTFILE] file [file ...]

An attempt to modify so that only the text of an html document is parsed and also allow the input/output of other charset encodings. The default is utf-8 if not specified on the command-line. I got it to work with either utf-8 or windows-1252.

[Jellby]

I would render the HTML in a browser, copy and paste in a text file, and extract the unique chars from there.

[pdurrant]

Originally Posted by Jellby

I would render the HTML in a browser, copy and paste in a text file, and extract the unique chars from there.

That's hard to automate and is overkill. If we accidentally include one or two characters that aren't in the rendered text, that's not a problem.

Now we need something that can process a TTF or OTF file and create a sub-set of the font.

[Doitsu]

BTW, Python-challenged ebook designers could simply compile an epub with KindlePreviewer/KindleGen and have a look at the detected Unicode ranges in the log file. For example, if you compile the book mentioned in roger64's post you'll see the following output:

Info(prcgen):I1045: Computing UNICODE ranges used in the book
Info(prcgen):I1046: Found UNICODE range: Basic Latin [20..7E]
Info(prcgen):I1046: Found UNICODE range: General Punctuation - Windows 1252 [2018..201A]
Info(prcgen):I1046: Found UNICODE range: Latin-1 Supplement [A0..FF]
Info(prcgen):I1046: Found UNICODE range: General Punctuation - other than Windows 1252 [2015..2017]
Info(prcgen):I1046: Found UNICODE range: Latin Extended-A [100..17F]
Info(prcgen):I1046: Found UNICODE range: Basic Greek [370..3FF]
Info(prcgen):I1046: Found UNICODE range: Greek Extended [1F00..1FFF]

Windows users could then use the MS Font Properties Extension to check the Unicode coverage of their fonts.

[Jellby]

Originally Posted by pdurrant

That's hard to automate and is overkill.

I agree, but it could be easier if Sigil makes the list, I've just created a suggestion for this.

[pdurrant]

Originally Posted by Doitsu

BTW, Python-challenged ebook designers could simply compile an epub with KindlePreviewer/KindleGen and have a look at the detected Unicode ranges in the log file.

I think that getting the ranges isn't fine grained enough. We're not wanting to check that our fonts cover the characters used, but to trim the fonts to cover only the characters used. Of course, making sure that all the needed characters are in the font will be part of this.

[Man Eating Duck]

Originally Posted by pdurrant

I think that getting the ranges isn't fine grained enough. We're not wanting to check that our fonts cover the characters used, but to trim the fonts to cover only the characters used. Of course, making sure that all the needed characters are in the font will be part of this.

Actually I am very interested in finding out if there are missing glyphs... Small squares in place of characters is a far larger problem than a few tens of Kilobytes in size IMO

I suspect that most methods of subsetting would also give you a "free" coverage check in the bargain.

[Freeshadow]

Originally Posted by pdurrant

Now we need something that can process a TTF or OTF file and create a sub-set of the font.

Fontforge can AFAIR be controlled by python scripts.