kobopatch - A new (WIP) patching system

[geek1011]

Continued from https://www.mobileread.com/forums/sh...77#post3686505

---

I'm working on a new patching system written in Go as a replacement for patch32lsb. The code is on GitHub here. It will have many more advantages for example:

• zlib support
• No need for external dependencies (like 7-zip)
• More flexibility (both in patch format and the actual patching)
• More consistent and readable patch format
  • New instructions like FindReplaceString (combines FindBaseAddress and ReplaceString)
  • Patch group checking
  • More
• All-in-one version
  • Will not need temporary files
  • Will be able to work with basically any OS
  • Can be embedded in other applications
  • Will preserve all file metadata (instead of just setting the permission to executable)
  • Possible web interface
  • Faster
• Unit tests (to be able to check for breakages)

I do not expect it to be ready and stable enough for general use until about a month. You can still try if you are ready for the possibility to need to wipe your sd card.


So far, I've finished a drop-in replacement for patch32lsb (with the old format) here which can be downloaded here. Please use this to test my patching system to make sure it produces the exact same results as the original patch32lsb. On Linux, this can be automated with the testallnickel and testalllibnickel scripts in the root of the repository. As far as I have tested, it seems to work well. The reason why it can be tested this way is because my patching system is modular, and the format is separated from the patching system itself.


I've also almost finished a working POC for the new all-in-one patcher. Here (minus README.md and oldpatches) is a sample of the layout, and here is a sample (which is subject to change) of the features of the new patch format.

This thread is old. See https://www.mobileread.com/forums/sh....php?p=3697837

[geek1011]

Here are some regexps I'm working on to convert the patches to my (WIP) new format (the output needs to be checked, as these don't cover edge cases):

Note: These need to be applied without case sensitivity.

replace_bytes:

Spoiler:

replace_bytes ?= ?([0-9A-Fa-f]+), ?([0-9A-Fa-f ]+), ?([0-9A-Fa-f ]+)$

to

  - ReplaceBytes: {Offset: 0x$1, FindH: $2, ReplaceH: $3}

replace_int:

Spoiler:

replace_int ?= ?([0-9A-Fa-f]+), ?([0-9-]+), ?([0-9-]+)$

to

  - ReplaceInt: {Offset: 0x$1, Find: $2, Replace: $3}

replace_float:

Spoiler:

replace_float ?= ?([0-9A-Fa-f]+), ([0-9.-]+), ?([0-9.-]+)$

to

  - ReplaceFloat: {Offset: 0x$1, Find: $2, Replace: $3}

base_address:

Spoiler:

base_address ?= ?([0-9A-Fa-f]+)

to

  - BaseAddress: 0x$1

patch_name:

Spoiler:

patch_name ?= ?`(.+?)`$

to

$1:

patch_enable:

Spoiler:

patch_enable ?= ?`(.+?)`$

to

  - Enabled: $1

find_base_address:

Spoiler:

find_base_address ?= ?`(.+?)`$

to

  - FindBaseAddressString: "$1"

Important: For patches which consist of strings of only bytes, use FindBaseAddressHex instead!

replace_string:

Spoiler:

replace_string ?= ?([0-9A-Fa-f]+), ?`((?:[^`\\]|\\.)*)`, ?`((?:[^`\\]|\\.)*)`$

to

  - ReplaceString: {Offset: 0x$1, Find: "$2", Replace: "$3"}

patch_group:

Spoiler:

# ?patch_group ?= ?`(.+?)`

to

  - PatchGroup: $1

patch:

Spoiler:

(<patch>|</patch>)

to

The description and cleanup needs to be done manually, and the output needs to be checked.

[geek1011]

reserved 1

[geek1011]

reserved 2

[geek1011]

Anything in bold is something I want more opinions about.

Originally Posted by jackie_w

As an occasional patch contributor I'll be happy to test your new system if I can do the testing on Windows. Is this possible at the moment?

Yes, but you will need to compare the output manually (the checksums).

Originally Posted by jackie_w

Also a comment about Offset/Find/Replace…
I like the new option which places each of these on its own line rather than combining into a single line. It should be less confusing for new users who want to tweak the defaults. For me the single line of the current system is what makes it more awkward to ensure that the before/after strings are the same length.

Yes. That is part of the flexibility of the new system. A patch can go in one line, multiple lines, or even a combination. It supports any feature YAML (the markup language) supports.

Originally Posted by jackie_w

I'm looking forward to being able to patch nickel zlib css in a more user-friendly way. I'm hoping it will make it possible for users to fine-tune them.

Yes. I'm still working out how to extract the strings efficiently, but I'm making progress so far. If @GeoffR has any tips, they would be useful. I am also looking for ideas on how everyone wants this feature to look (simple find/replace, find by CSS ID, etc).

Originally Posted by oren64

@geek1011 it works ok, what the difference between the old and new file.

My version has more built-in checks and will end up having it's own format and become a complete patching system. The drop-in binary was only for testing and migration. Also, I'll be writing a converter to convert from the old format to the new one. How many people want this? I could also just add compatibility with the old format, but parsing it is error-prone due to its ambiguity.

Originally Posted by jackie_w

I should have been more specific in my question. I saw a couple of linux files (testalllibnickel.sh and testallnickel.sh) which I thought (perhaps wrongly) were needed to check old-result vs. new-result.

You were right (partially). Those scripts just automate checking the differences between the old and the new patcher.

Originally Posted by dmapr

- FindBaseAddress: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n"
- ReplaceString: {Offset: 0x0, Find: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n", Replace: "ReadingFooter {min-height:15px;max-height:15px;}\n"}

One thing that stands out in some of these examples is repeating the search string in the FindBaseAddress and ReplaceString instructions. Is it possible to write the above as

- FindBaseAddress: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n"
- ReplaceString: {Offset: 0x0, Replace: "ReadingFooter {min-height:15px;max-height:15px;}\n"}

omitting the Find parameter and the default being the value passed to the FindBaseAddress?

That seems like a good idea. I could implement that as a FindReplaceString instruction which would combine the two. What do you think of that?

[geek1011]

I've implemented a new instruction for the new patch format: FindReplaceString.

An example:
Instead of:

- FindBaseAddress: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n"
- ReplaceString: {Offset: 0x0, Find: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n", Replace: "ReadingFooter {min-height:15px;max-height:15px;}\n"}

You can do:

- FindReplaceString: {Find: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n", Replace: "ReadingFooter {min-height:15px;max-height:15px;}\n"}

[geek1011]

I've now implemented PatchGroup checking in the new patch format. kobopatch will give an error if more than 1 enabled patch has the same PatchGroup.

[jackie_w]

Originally Posted by geek1011

Yes, but you will need to compare the output manually (the checksums).

I can do checksums easily enough. Why didn't I think of that

Originally Posted by geek1011

Yes. I'm still working out how to extract the strings efficiently, but I'm making progress so far. If @GeoffR has any tips, they would be useful. I am also looking for ideas on how everyone wants this feature to look (simple find/replace, find by CSS ID, etc).

Some comments based on my experience of using the pipcat-method...

I'm not sure CSS Id (e.g /* found: 116 (zlib) pos: 45e7e8 */) is any help because the id number (116 in my example) changes with every firmware, even if the CSS content itself hasn't changed.

I believe it's always possible to define a unique string to identify the required CSS stream, (e.g. #N3TableOfContentsWidget[qApp_deviceIsTrilogy=true]), but you can't rely on being able to find a unique string which starts at the first character in the stream.

[jackie_w]

Can you also consider creating a physical output log .txt file. When users have problems, it might be helpful if it's easy for them to attach their error log when they ask for help. Or are you planning to make the new system so easy no-one ever makes silly mistakes

[geek1011]

I've now implemented a Description option in the new patch format. This makes it look a lot cleaner.

Here is an example patch using Description, FindReplaceString, and PatchGroup:

# Versions: 4.6.9960 - 4.8.10956+
Disable Reading Footer:
  - Enabled: true
  - PatchGroup: Test Patch Group
  - Description: |
      Removes the reading footer for both ePub and KePub books, leaving a 15px
      bottom margin which matches the 15px built-in top margin of ePub books.
  - FindReplaceString: {Find: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n", Replace: "ReadingFooter {min-height:15px;max-height:15px;}\n"}
  - FindReplaceString: {Find: "#caption[qApp_deviceIsTrilogy=true] {\n  font-size: 19px;\n}\n", Replace: "#caption {font-size:1px;color:transparent;}\n\0"}

[Terisa de morgan]

Originally Posted by geek1011

Also, I'll be writing a converter to convert from the old format to the new one. How many people want this? I could also just add compatibility with the old format, but parsing it is error-prone due to its ambiguity.

A converter would be nice... for Geoff and oren_64. The rest of us copy and paste text at best. I think, in the long term, it will be simpler to maintain only a format for the patches.

[geek1011]

Originally Posted by jackie_w

I'm not sure CSS Id (e.g /* found: 116 (zlib) pos: 45e7e8 */) is any help because the id number (116 in my example) changes with every firmware, even if the CSS content itself hasn't changed.

I can have a FindZlibBaseAddress instruction which can take either: a checksum of the CSS, the CSS Id, or a unique string anywhere in the CSS.

Also, should I make an option to dump the CSS?

Originally Posted by jackie_w

Can you also consider creating a physical output log .txt file. When users have problems, it might be helpful if it's easy for them to attach their error log when they ask for help. Or are you planning to make the new system so easy no-one ever makes silly mistakes

I'm going to do the in the new patcher. Remember, the drop-in replacement is only for testing without having to rewrite all the patches and for me to try implementing a parser for the old format. And yes, I'll try to make this as impossible as possible to make it possible to make mistakes.

[geek1011]

Originally Posted by Terisa de morgan

A converter would be nice... for Geoff and oren_64. The rest of us copy and paste text at best. I think, in the long term, it will be simpler to maintain only a format for the patches.

Yes. The converter will not be perfect, and will need manual tweaking after. Also, I think it may be easier after Geoff and Oren get used to the new format, since my format has more flexibility (such as being able to use unicode escapes, being able to use int offsets, and the validation).

For copying and pasting, an advantage of my format is long patches (such as zlib ones) can be made into 1 line if needed. Here is an example:

Short:

{'Disable Reading Footer': [{Enabled: true}, {Description: "Removes the reading footer for both ePub and KePub books, leaving a 15px\nbottom margin which matches the 15px built-in top margin of ePub books.\n"}, {FindReplaceString: {Find: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n", Replace: "ReadingFooter {min-height:15px;max-height:15px;}\n"}}, {FindReplaceString: {Find: "#caption[qApp_deviceIsTrilogy=true] {\n  font-size: 19px;\n}\n", Replace: "#caption {font-size:1px;color:transparent;}\n\0"}}]}

Readable:

# Versions: 4.6.9960 - 4.8.10956+
Disable Reading Footer:
  - Enabled: true
  - Description: |
      Removes the reading footer for both ePub and KePub books, leaving a 15px
      bottom margin which matches the 15px built-in top margin of ePub books.
  - FindReplaceString: {Find: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n", Replace: "ReadingFooter {min-height:15px;max-height:15px;}\n"}
  - FindReplaceString: {Find: "#caption[qApp_deviceIsTrilogy=true] {\n  font-size: 19px;\n}\n", Replace: "#caption {font-size:1px;color:transparent;}\n\0"}

Long:

# Versions: 4.6.9960 - 4.8.10956+
Disable Reading Footer:
  - Enabled: true
  - Description: |
      Removes the reading footer for both ePub and KePub books, leaving a 15px
      bottom margin which matches the 15px built-in top margin of ePub books.

  - FindReplaceString: 
    Find: "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n"
    Replace: "ReadingFooter {min-height:15px;max-height:15px;}\n"

  - FindReplaceString: 
    Find: "#caption[qApp_deviceIsTrilogy=true] {\n  font-size: 19px;\n}\n"
    Replace: "#caption {font-size:1px;color:transparent;}\n\0"

[jackie_w]

Originally Posted by geek1011

I can have a FindZlibBaseAddress instruction which can take either: a checksum of the CSS, the CSS Id, or a unique string anywhere in the CSS.

I'm going to assume that your aim is for the creator/maintainer of each patch to have as little work to do as possible (ideally none at all) for each new firmware. If so the 3rd option is the one least likely to change between firmwares, assuming one chooses the unique string wisely, of course.

Originally Posted by geek1011

Also, should I make an option to dump the CSS?

Yes - for those who create/maintain patches. Not necessary for the average user, though.

[geek1011]

Originally Posted by jackie_w

I'm going to assume that your aim is for the creator/maintainer of each patch to have as little work to do as possible (ideally none at all) for each new firmware. If so the 3rd option is the one least likely to change between firmwares, assuming one chooses the unique string wisely, of course.


Yes - for those who create/maintain patches. Not necessary for the average user, though.

OK. And yes, I'm trying to make the patching process as easy as possible for everyone. The only thing is I'm not going to make a feature to put the css back. For that, you'll need to make a patch and apply that instead.

Also, can someone who knows about the internals of the CSS patches like @oren64 or @GeoffR tell me a bit about how to extract it? I've been getting stuck on properly finding the right Zlib streams.