User column based on part of "Comments" ?

[chaley]

Originally Posted by pchrist7

How do you debug ? Just looking, trying, etc. ?

Primitively.

I use print statements. Sometimes I toss in stack tracebacks to see how I got to a particular place, but that isn't useful for template functions. Printing does work fine in template functions.

Just in case you don't know: if you open a command box (I think you are running windows?), and run calibre-debug -g, then the debug output (print statements and the like) will show up there. For me, this is more convenient (much faster) than restarting calibre in debug mode and waiting to see the box of output that shows up when you close calibre.

[pchrist7]

Originally Posted by chaley

Primitively.

I use print statements. Sometimes I toss in stack tracebacks to see how I got to a particular place, but that isn't useful for template functions. Printing does work fine in template functions.

Just in case you don't know: if you open a command box (I think you are running windows?), and run calibre-debug -g, then the debug output (print statements and the like) will show up there. For me, this is more convenient (much faster) than restarting calibre in debug mode and waiting to see the box of output that shows up when you close calibre.

Ahh - never thought of that.
Will try this.
Any way to "set -x" like in most unix shells ?

[pchrist7]

Originally Posted by beckywc

What I want the custom column to show is the date read if > 95%. If the book shows >0% but < 95% I want the colunn to show reading and if the book is 0% or unopened not read.

Hi beckywc.
With great help from chaley, once again, I think I have what you want.
In exchange I want your help to create a howto for the sticky thread about custom columns
Please look closely at the instructions, grammar, punctuations etc.
I want this to be as precise as possible.
Also, do you understand the Documentation part in the Template function ?
Please suggest corrections, clarifications.
So please try the following instructions, to the letter, to see if I got it all right:
And of course, please post all comments to this thread.


Howto:

Do you have a Kindle ?
Did you ever want Calibre to show "Read Status" of your books in a column ?
Automagically ?

Then the following might for you as well.

You must be running Calibre 0.7.40 or newer to use this.
Also you must use "Fetch annotations" from you Kindle.

The following example is based on the following Kindler requirements:
"What I want is a custom column to show the "Last Opened" Date if Last location read > 95%.
If the book shows >0% but < 95%, I want the column to show "Reading"
And if the book is 0% or unopened the column should show "Not Read".


How-to "Fetch annotations" from you Kindle:
Start Calibre,
Connect your Kindle, wait a bit,
Click an the arrow just to the right of the "Send to Device" icon,
choose "Fetch Annotations"
and the Kindle "Book status" is added to the Calibre book details, in the comments.

You should Fetch fresh Annotations everytime you connect your Kindle to get updated information.

This is just an example of what I could with my K3:

25-12-2010
Last Page Read: Location 7365 (68%)
Location 4603 • Bookmark
Location 4942 • Bookmark
Location 4980 • Bookmark
Location 5984 • Bookmark
Location 7363 • Bookmark


The best/fastest way, but hardest way, to get the needed information into a custom column, is to use Python Template functions.
Sounds scary, but don't worry. Copy/paste is your helper

Simply do as follows:

1) go to Preferences -> Advanced -> Template Functions.

2) in the Function box type: kindle_read_status

3) in the Arg count box type: 4

4) in the Documentation box, paste the following:

Check if the associated field contains a kindle annotation for percent read. If not, return no_page_read_str. If so, then compare that percent against is_read_pct. If the val is larger, then return date, if the val is >0 and < is_read_pct return is_reading_str. Otherwise return is_not_read_str. 

One usage: {comments:kindle_read_status(95,Not Read,Reading)}
( use the above when defining your custom column )
95 goes into variable is_read_pct
Not Read goes into variable no_page_read_str
Reading goes into variable is_reading_str

Feel free to change the %, the test strings to suit your needs.
Do this in the custom column definition.

5) In the Program Code box, paste the following:

def evaluate(self, formatter, kwargs, mi, locals, val, is_read_pct,  
				is_reading_str, no_page_read_str):
	try:
		test_val = int(is_read_pct)
	except:
		return 'is_read_pct is not a number'

	import re
	mg = re.match('.*\s(\d+[-/]\d+[-/]\d\d\d\d).*?Last Page Read: Location \d+ \((\d+)%\)', val, re.I + re.DOTALL);
	if mg is None:
		return no_page_read_str
	date = mg.group(1)
	pct = mg.group(2)
	try:
		f = int(pct)
		if f > test_val:
			return date
		elif f > 0:
			return is_reading_str + ': ' + pct + '%'
	except:
		pass
	return no_page_read_str

6) press the Create button, then the Apply button.

7) Still in Preferences, Choose Add your own columns

8) Choose Add custom column bar at the bottom

9) enter the following into your custom column:
Lookup name: read_status
Column Heading: Read Status
Column type: Column built from other columns
Template:

{comments:kindle_read_status(95,Reading,Not Read)}

10) Choose Apply

11) Click Ok to the Restart Warning

12) Shutdown Calibre

13) Start Calibre

If you followed instructions minutely, you should have a new column showing the "Read Status" for your books.

If you like this functionality, please remember to send some Karma to chaley

https://www.mobileread.com/forums/rep....php?p=1351864

He (?) has been absolutely fantastic and helpfull with this.

If this fails, please try one more time before posting to this thread.
This has been tested by several Kindlers succesfully.
Enjoy

Edit: Using the code formatting of the Template function documentation, looses all line breaks during copy/paste. Any suggestions ?
Also, need help with the "chaley karma" link

[chaley]

Originally Posted by pchrist7

Ahh - never thought of that.
Will try this.
Any way to "set -x" like in most unix shells ?

First, thank you for taking the lead on getting a user-friendly version of this set up!

Now to your question: What are you trying to do? When you run a .bat file in a windows box, you get '-x'-like behavior for the commands in the file. However, the commands are extremely uninteresting, so this can't be what you are thinking of.

If you are thinking of a python execution trace, then yes, but it is somewhat complex. The following example shows one way to do it. What you want in 'evaluate' goes into 'myfunc', and 'evaluate' becomes the controller that sets up the trace. This example runs for me. Note that when looking at the output, the line number of the myfunc line is 4, not 1, because there is stuff you are not seeing.

def myfunc(self, formatter, kwargs, mi, locals, val, is_read_pct, is_read_str, 
				is_not_read_str, no_page_read_str):
	try:
		test_val = int(is_read_pct)
	except:
		return 'is_read_pct is not a number'
	print 'here'
	import re
	mg = re.match('.*last page read: location \d+ \((\d+)%\)', val, re.I + re.DOTALL);
	if mg is None:
		return no_page_read_str
	print 'here2'
	try:
		f = int(mg.group(1))
		if f >= test_val:
			return is_read_str
		return is_not_read_str
	except:
		pass
	return no_page_read_str

def evaluate(self, *args):

	import linecache, inspect, sys

	def traceit(frame, event, arg):
		if event == 'line':
			lineno = frame.f_lineno
			if '__file__' in frame.f_globals:
				filename = frame.f_globals['__file__']
				if (filename.endswith('.pyc') or
					filename.endswith('.pyo')):
					filename = filename[:-1]
				name = frame.f_globals['__name__']
				line = linecache.getline(filename, lineno)
			else:
				name = '[unknown]'
				try:
					src = inspect.getsourcelines(frame)
					line = src[lineno]
				except IOError:
					line = 'Unknown code named [%s].  VM instruction #%d' % \
						(frame.f_code.co_name, frame.f_lasti)
			print '%s:%s: %s' % (name, lineno, line.rstrip())
		return traceit

	sys.settrace(traceit)
	try:
		rv = self.myfunc(*args)
	except:
		rv = 'EXCEPTION'
	sys.settrace(None)
	return rv

Acknowledgment: code for the traceit function from http://www.friday.com/bbum/2005/10/2...hon-execution/

[pchrist7]

Originally Posted by chaley

First, thank you for taking the lead on getting a user-friendly version of this set up!

Well, I DID promise to do it, right ?

Now to your question: What are you trying to do? When you run a .bat file in a windows box, you get '-x'-like behavior for the commands in the file. However, the commands are extremely uninteresting, so this can't be what you are thinking of.

In a unix shell script you use #!/usr/bin/ksh -x as first line in script, or any where else in script use set -x
Then you'll see, line by line, how the scripts is executed, what value is assigned to variables, comparisons, etc.
I was more looking for a "python -x " thingy that could do something similar.
Forgive my ignorance, but sometimes wishes DO come true. ( I heard )

If you are thinking of a python execution trace, then yes, but it is somewhat complex. The following example shows one way to do it. What you want in 'evaluate' goes into 'myfunc', and 'evaluate' becomes the controller that sets up the trace. This example runs for me. Note that when looking at the output, the line number of the myfunc line is 4, not 1, because there is stuff you are not seeing.

Thanks. Will give it a try sometime.
Still working, but now from home

def myfunc(self, formatter, kwargs, mi, locals, val, is_read_pct, is_read_str, 
				is_not_read_str, no_page_read_str):
	try:
		test_val = int(is_read_pct)
	except:
		return 'is_read_pct is not a number'
	print 'here'
	import re
	mg = re.match('.*last page read: location \d+ \((\d+)%\)', val, re.I + re.DOTALL);
	if mg is None:
		return no_page_read_str
	print 'here2'
	try:
		f = int(mg.group(1))
		if f >= test_val:
			return is_read_str
		return is_not_read_str
	except:
		pass
	return no_page_read_str

def evaluate(self, *args):

	import linecache, inspect, sys

	def traceit(frame, event, arg):
		if event == 'line':
			lineno = frame.f_lineno
			if '__file__' in frame.f_globals:
				filename = frame.f_globals['__file__']
				if (filename.endswith('.pyc') or
					filename.endswith('.pyo')):
					filename = filename[:-1]
				name = frame.f_globals['__name__']
				line = linecache.getline(filename, lineno)
			else:
				name = '[unknown]'
				try:
					src = inspect.getsourcelines(frame)
					line = src[lineno]
				except IOError:
					line = 'Unknown code named [%s].  VM instruction #%d' % \
						(frame.f_code.co_name, frame.f_lasti)
			print '%s:%s: %s' % (name, lineno, line.rstrip())
		return traceit

	sys.settrace(traceit)
	try:
		rv = self.myfunc(*args)
	except:
		rv = 'EXCEPTION'
	sys.settrace(None)
	return rv

Acknowledgment: code for the traceit function from http://www.friday.com/bbum/2005/10/2...hon-execution/[/QUOTE]

Will give it a couple of test runs tomorrow
BTW: You have any tips for my latest added "edit" comments ?
Hoping beckywc will try my howto during the weekend.
Did 3 test runs, in 3 different Calibre libs.
My main lib with 1k books works fine, no performance penalty I can detect.
Have a nice weekend

Edit: I forgot to mention that I really like your thinking :
return is_reading_str + ': ' + pct + '%'

Thats is NEAT !
Never thought of that myself.
Let's see if bekywc and others choke, or love it :-)

[beckywc]

Originally Posted by pchrist7

Hi beckywc.
With great help from chaley, once again, I think I have what you want.
In exchange I want your help to create a howto for the sticky thread about custom columns
Please look closely at the instructions, grammar, punctuations etc.
I want this to be as precise as possible.
Also, do you understand the Documentation part in the Template function ?
Please suggest corrections, clarifications.
So please try the following instructions, to the letter, to see if I got it all right:
And of course, please post all comments to this thread.


Howto:

Do you have a Kindle ?
Did you ever want Calibre to show "Read Status" of your books in a column ?
Automagically ?

Then the following might for you as well.

You must be running Calibre 0.7.40 or newer to use this.
Also you must use "Fetch annotations" from you Kindle.

The following example is based on the following Kindler requirements:
"What I want is a custom column to show the "Last Opened" Date if Last location read > 95%.
If the book shows >0% but < 95%, I want the column to show "Reading"
And if the book is 0% or unopened the column should show "Not Read".


How-to "Fetch annotations" from you Kindle:
Start Calibre,
Connect your Kindle, wait a bit,
Click an the arrow just to the right of the "Send to Device" icon,
choose "Fetch Annotations"
and the Kindle "Book status" is added to the Calibre book details, in the comments.

This is just an example of what I could with my K3:

25-12-2010
Last Page Read: Location 7365 (68%)
Location 4603 • Bookmark
Location 4942 • Bookmark
Location 4980 • Bookmark
Location 5984 • Bookmark
Location 7363 • Bookmark


The best/fastest way, but hardest way, to get the needed information into a custom column, is to use Python Template functions.
Sounds scary, but don't worry. Copy/paste is your helper

Simply do as follows:

1) go to Preferences -> Advanced -> Template Functions.

2) in the Function box type: kindle_read_status

3) in the Arg count box type: 4

4) in the Documentation box, paste the following:

Check if the associated field contains a kindle annotation for percent read. If not, return no_page_read_str. If so, then compare that percent against is_read_pct. If the val is larger, then return date, if the val is >0 and < is_read_pct return is_reading_str. Otherwise return is_not_read_str. 

One usage: {comments:kindle_read_status(95,Not Read,Reading)}
( use the above when defining your custom column )
95 goes into variable is_read_pct
Not Read goes into variable no_page_read_str
Reading goes into variable is_reading_str

Feel free to change the %, the test strings to suit your needs.
Do this in the custom column definition.

5) In the Program Code box, paste the following:

def evaluate(self, formatter, kwargs, mi, locals, val, is_read_pct,  
				is_reading_str, no_page_read_str):
	try:
		test_val = int(is_read_pct)
	except:
		return 'is_read_pct is not a number'

	import re
	mg = re.match('.*\s(\d+[-/]\d+[-/]\d\d\d\d).*?Last Page Read: Location \d+ \((\d+)%\)', val, re.I + re.DOTALL);
	if mg is None:
		return no_page_read_str
	date = mg.group(1)
	pct = mg.group(2)
	try:
		f = int(pct)
		if f > test_val:
			return date
		elif f > 0:
			return is_reading_str + ': ' + pct + '%'
	except:
		pass
	return no_page_read_str

6) press the Create button, then the Apply button.

7) Still in Preferences, Choose Add your own columns

8) Choose Add custom column bar at the bottom

9) enter the following into your custom column:
Lookup name: read_status
Column Heading: Read Status
Column type: Column built from other columns
Template:

{comments:kindle_read_status(95,Reading,Not Read)}

10) Choose Apply

11) Click Ok to the Restart Warning

12) Shutdown Calibre

13) Start Calibre

If you followed instructions minutely, you should have a new column showing the "Read Status" for your books.

If you like this functionality, please remember to send some Karma to chaley

https://www.mobileread.com/forums/rep....php?p=1351864

He (?) has been absolutely fantastic and helpfull with this.

If this fails, please try one more time before posting to this thread.
This has been tested by several Kindlers succesfully.
Enjoy

Edit: Using the code formatting of the Template function documentation, looses all line breaks during copy/paste. Any suggestions ?
Also, need help with the "chaley karma" link

Originally Posted by pchrist7

Well, I DID promise to do it, right ?


In a unix shell script you use #!/usr/bin/ksh -x as first line in script, or any where else in script use set -x
Then you'll see, line by line, how the scripts is executed, what value is assigned to variables, comparisons, etc.
I was more looking for a "python -x " thingy that could do something similar.
Forgive my ignorance, but sometimes wishes DO come true. ( I heard )


Thanks. Will give it a try sometime.
Still working, but now from home

def myfunc(self, formatter, kwargs, mi, locals, val, is_read_pct, is_read_str, 
				is_not_read_str, no_page_read_str):
	try:
		test_val = int(is_read_pct)
	except:
		return 'is_read_pct is not a number'
	print 'here'
	import re
	mg = re.match('.*last page read: location \d+ \((\d+)%\)', val, re.I + re.DOTALL);
	if mg is None:
		return no_page_read_str
	print 'here2'
	try:
		f = int(mg.group(1))
		if f >= test_val:
			return is_read_str
		return is_not_read_str
	except:
		pass
	return no_page_read_str

def evaluate(self, *args):

	import linecache, inspect, sys

	def traceit(frame, event, arg):
		if event == 'line':
			lineno = frame.f_lineno
			if '__file__' in frame.f_globals:
				filename = frame.f_globals['__file__']
				if (filename.endswith('.pyc') or
					filename.endswith('.pyo')):
					filename = filename[:-1]
				name = frame.f_globals['__name__']
				line = linecache.getline(filename, lineno)
			else:
				name = '[unknown]'
				try:
					src = inspect.getsourcelines(frame)
					line = src[lineno]
				except IOError:
					line = 'Unknown code named [%s].  VM instruction #%d' % \
						(frame.f_code.co_name, frame.f_lasti)
			print '%s:%s: %s' % (name, lineno, line.rstrip())
		return traceit

	sys.settrace(traceit)
	try:
		rv = self.myfunc(*args)
	except:
		rv = 'EXCEPTION'
	sys.settrace(None)
	return rv

Acknowledgment: code for the traceit function from http://www.friday.com/bbum/2005/10/2...hon-execution/

Will give it a couple of test runs tomorrow
BTW: You have any tips for my latest added "edit" comments ?
Hoping beckywc will try my howto during the weekend.
Did 3 test runs, in 3 different Calibre libs.
My main lib with 1k books works fine, no performance penalty I can detect.
Have a nice weekend

Edit: I forgot to mention that I really like your thinking :
return is_reading_str + ': ' + pct + '%'

Thats is NEAT !
Never thought of that myself.
Let's see if bekywc and others choke, or love it :-)[/QUOTE]

I get EXCEPTION for reading in the column though your first one worked.

[pchrist7]

I get EXCEPTION for reading in the column though your first one worked.

Hi bechywc
Thanks for testing.
Please post what worked, and what didn't work
Need more info, and sleep, so let's see tomorrow
Cheers,

[beckywc]

In the custom column the date appears for those books that have been read and not read appears for those that have not been opened or read. But EXCEPTION is in the column for those that I have started to read.

[chaley]

Originally Posted by beckywc

In the custom column the date appears for those books that have been read and not read appears for those that have not been opened or read. But EXCEPTION is in the column for those that I have started to read.

Did you copy the code from pchrist7's post (#63) or mine (#64)? The appearance of EXCEPTION implies it is mine. It must be his, as mine doesn't implement any of his changes, but is demonstrating something else.

[beckywc]

Sorry I did copy the wrong code. The code from #63 works.

This is so cool. Thanks

[pchrist7]

Originally Posted by beckywc

Sorry I did copy the wrong code. The code from #63 works.

This is so cool. Thanks

Hi beckywc
You're welcome.
Any comments, suggestions to the HOWTO ?

[pchrist7]

Originally Posted by chaley

If you are thinking of a python execution trace, then yes, but it is somewhat complex. The following example shows one way to do it. What you want in 'evaluate' goes into 'myfunc', and 'evaluate' becomes the controller that sets up the trace. This example runs for me. Note that when looking at the output, the line number of the myfunc line is 4, not 1, because there is stuff you are not seeing.

Hi Chaley
Had a change to fool around with your code.
For a start I think I'll stay with the print method.
Printing variables, locations in code etc.
One thing though - thought I could print mg like this
print "mg: %s" % (mg)
If mg is empty "mg: None" is printed as expected.
If mg has contents, "mg: <_sre.SRE_Match object at 0x0CFCF2A0>"
I had expected the text content of mg, so a bit confused.

[beckywc]

Originally Posted by pchrist7

Hi beckywc
You're welcome.
Any comments, suggestions to the HOWTO ?

Looks fine to me.

[chaley]

Originally Posted by pchrist7

Had a change to fool around with your code.
For a start I think I'll stay with the print method.

No surprise, that. The trace would be useful in very limited circumstances. In most, it is data overload.

One thing though - thought I could print mg like this
print "mg: %s" % (mg)
If mg is empty "mg: None" is printed as expected.
If mg has contents, "mg: <_sre.SRE_Match object at 0x0CFCF2A0>"
I had expected the text content of mg, so a bit confused.

You have stumbled across a python'ism. Printing object instances requires class-level support. The class must include some special methods that return a string representation of an instance of the class. SR match objects don't, so you need to make do with its API. For example, you can print mg.groups(), which will give you a list of all the matched groups. mg.group(0) will give you the entire matched string.

[pchrist7]

Originally Posted by chaley

You have stumbled across a python'ism. Printing object instances requires class-level support. The class must include some special methods that return a string representation of an instance of the class. SR match objects don't, so you need to make do with its API. For example, you can print mg.groups(), which will give you a list of all the matched groups. mg.group(0) will give you the entire matched string.

Brilliant ! Thank you so much !
That will make this a lot easier in times to come ;-D