MobileRead Forums - View Single Post - KPV: a PDF reader for Kindle, based on muPDF, GPLv3

dave2008 · 02-25-2012, 09:28 AM

Hey guys,

When I am working on the jump stack feature, I found it not so easy to save the stack into book specific configuration files with current settings module. I think the "jump stack" information is best represented as a nested table, but the setting module only support (key, value) bindings. Thus I cannot find an elegant way to save nested table with the settings module. (I only came up with a some how dirty hack)

So I tried the table Serialization approach and did a benchmark between it and the settings module, which uses sqlite3.

As first, I thought since the settings module uses sqlite3, it should be faster. But the result turned out that it is not true:

Code:

  $ ./benchmark_pickle.lua                                              git:master [cb982f5] modified untracked
Time for sqlite3 approach: 2.05
Time for pickle approach: 0.19

The Serialization approach, using pickle module, is about 10 times faster though it handles more data than the sqlite3 approach. You can check out the benchmark code from following link:
https://github.com/houqp/kindlepdfvi...ark_pickle.lua

So I suggest using Serialization approach in the settings module instead of sqlite3 for four reasons:

1. Serialization approach is faster, as my test shows
2. Serialization approach generates human readable text stream instead of binary data, which is easier to maintain and modify by humans if needed.
3. Serialization approach make the code cleaner, you can dump the whole table with one function call.
4. For small data, Serialization approach dumps smaller setting file (395 bytes vs 3.0K in my test case). Haven't tested with large data tough.

I only tested the pickle module, but there are many other implementations which might be even faster:
http://lua-users.org/wiki/TableSerialization

So, any suggestions or feedbacks for this?

02-25-2012, 09:28 AM	#73
dave2008 Addict Posts: 251 Karma: 183457 Join Date: Jan 2012 Device: k3G, KDXG, AuraHD	Hey guys, When I am working on the jump stack feature, I found it not so easy to save the stack into book specific configuration files with current settings module. I think the "jump stack" information is best represented as a nested table, but the setting module only support (key, value) bindings. Thus I cannot find an elegant way to save nested table with the settings module. (I only came up with a some how dirty hack) So I tried the table Serialization approach and did a benchmark between it and the settings module, which uses sqlite3. As first, I thought since the settings module uses sqlite3, it should be faster. But the result turned out that it is not true: Code: $ ./benchmark_pickle.lua git:master [cb982f5] modified untracked Time for sqlite3 approach: 2.05 Time for pickle approach: 0.19 The Serialization approach, using pickle module, is about 10 times faster though it handles more data than the sqlite3 approach. You can check out the benchmark code from following link: https://github.com/houqp/kindlepdfvi...ark_pickle.lua So I suggest using Serialization approach in the settings module instead of sqlite3 for four reasons: 1. Serialization approach is faster, as my test shows 2. Serialization approach generates human readable text stream instead of binary data, which is easier to maintain and modify by humans if needed. 3. Serialization approach make the code cleaner, you can dump the whole table with one function call. 4. For small data, Serialization approach dumps smaller setting file (395 bytes vs 3.0K in my test case). Haven't tested with large data tough. I only tested the pickle module, but there are many other implementations which might be even faster: http://lua-users.org/wiki/TableSerialization So, any suggestions or feedbacks for this? Last edited by dave2008; 02-25-2012 at 09:53 PM.