kobopatch - A new (WIP) patching system

[jackie_w]

Originally Posted by geek1011

The only thing is I'm not going to make a feature to put the css back. For that, you'll need to make a patch and apply that instead.

Is there something I'm not understanding ?

Let me outline my own problem with the current pipcat-plus-makepatch method which adds an unreadable, untweakable blackbox hexcode patch to GeoffR's nickel.patch file.

A hypothetical new patch...
I make a patch to a CSS stream which changes the GUI booklist Title to the Avenir font in bold and not italic. It will probably require me to also change the font-size. I use makepatch and publish it.

Everybody loves it (I did say this was hypothetical) or rather the idea of it, except
... I'd like to use Bookerly, not bold, at a different font-size
... no, I like the new bold but I preferred the old Georgia
... no, I want Bookerly italic
... etc etc

How many makepatch's do I need to create to make everyone happy?

What is desirable for a new approach to patching zlib CSS would be the ability to patch a zlib in the same way that the nozlib `Custom reading footer style` patch allows, i.e. the user can tweak one or more variables all for the same CSS stream.

Other opinions are also available...

[geek1011]

Originally Posted by jackie_w

Is there something I'm not understanding ?

Let me outline my own problem with the current pipcat-plus-makepatch method which adds an unreadable, untweakable blackbox hexcode patch to GeoffR's nickel.patch file.

A hypothetical new patch...
I make a patch to a CSS stream which changes the GUI booklist Title to the Avenir font in bold and not italic. It will probably require me to also change the font-size. I use makepatch and publish it.

Everybody loves it (I did say this was hypothetical) or rather the idea of it, except
... I'd like to use Bookerly, not bold, at a different font-size
... no, I like the new bold but I preferred the old Georgia
... no, I want Bookerly italic
... etc etc

How many makepatch's do I need to create to make everyone happy?

What is desirable for a new approach to patching zlib CSS would be the ability to patch a zlib in the same way that the nozlib `Custom reading footer style` patch allows, i.e. the user can tweak one or more variables all for the same CSS stream.

Other opinions are also available...

That's actually a great idea.

I'll probably add a config instruction where users can set variables and they can be substituted into any other instruction. What do all of you think of that?

[boriar]

@geek1011
Why not include a variable for each individual patch indicating which file should be applied?
That way one can place all the patches in a single file without worrying about which one to include it in.
If there are patches with two parts that must be used at the same time in two files, it must be indicated in another variable (or reuse the one of "group of patches") so that both are applied. Or better yet, in the same patch allow strings to be applied in one file and strings to be applied to another.
I hope I have explained you well

[geek1011]

Originally Posted by boriar

@geek1011
Why not include a variable for each individual patch indicating which file should be applied?
That way one can place all the patches in a single file without worrying about which one to include it in.
If there are patches with two parts that must be used at the same time in two files, it must be indicated in another variable (or reuse the one of "group of patches") so that both are applied. Or better yet, in the same patch allow strings to be applied in one file and strings to be applied to another.
I hope I have explained you well

Hmm. That's a good idea, but I'm not yet sure how I'd go about implementing it efficiently. I'll look into it later.

[geek1011]

I've released the first alpha version of kobopatch (the all-in-one version). The instructions for testing it out are in the GitHub release: https://github.com/geek1011/kobopatc...ses/tag/v0.4.0

Please try it and tell me what you think.

BTW, I'm working on a backwards-compatible shorthand version for the patches to make it less tedious to write the byte arrays.

[oren64]

If GeoffR will use the new patch32lsb file it will solve the double formatting problem, this why I think you should keep the old fromat.

@geek1011 you might want to look at Modifications for bad eyes it patch zlib streams.

[geek1011]

Originally Posted by oren64

If GeoffR will use the new patch32lsb file it will solve the double formatting problem, this why I think you should keep the old fromat.

@geek1011 you might want to look at Modifications for bad eyes it patch zlib streams.

What double formatting problem? I've never heard of it.

Also, the old format is really hard to parse correctly. It takes over 3x as much code, but it is still kind of unreliable (especially the string parsing). It also does not do things like check patch groups. I also cannot easily extend the old one with more complex instructions such as the CSS ones and the variable substitution ones I mentioned above. I've listed some of the advantages of the new format above.

[jackie_w]

Originally Posted by geek1011

Also, can someone who knows about the internals of the CSS patches like @oren64 or @GeoffR tell me a bit about how to extract it? I've been getting stuck on properly finding the right Zlib streams.

I'm not sure whether I'm answering the right question here but ...

Currently the initial CSS extract is done using pipcat's Python extract.py which calls css_streams.py. If you haven't already you can find all the code in this thread. Personally, I use his v3 code in post #8 (rather than v4 which also accommodates those who need tshering's Mods for Bad Eyes).

css_streams.py is where you'll find the extract algorithms but be aware that it has a small bug which will miss consecutive nozlib CSS streams if the second one starts immediately after the first. It's easily fixed by finding the line

pos += 4 + len_stream

and changing it to

pos += 3 + len_stream

[geek1011]

Originally Posted by jackie_w

I'm not sure whether I'm answering the right question here but ...

Currently the initial CSS extract is done using pipcat's Python extract.py which calls css_streams.py. If you haven't already you can find all the code in this thread. Personally, I use his v3 code in post #8 (rather than v4 which also accommodates those who need tshering's Mods for Bad Eyes).

css_streams.py is where you'll find the extract algorithms but be aware that it has a small bug which will miss consecutive nozlib CSS streams if the second one starts immediately after the first. It's easily fixed by finding the line

pos += 4 + len_stream

and changing it to

pos += 3 + len_stream

OK, thanks. That tells me about 25% of what I want to know and explains a few things.

BTW, right now, I'm working on converting and testing all the libnickel patches.

[oren64]

Originally Posted by geek1011

I've released the first alpha version of kobopatch (the all-in-one version). The instructions for testing it out are in the GitHub release: https://github.com/geek1011/kobopatc...ses/tag/v0.4.0

Please try it and tell me what you think.

BTW, I'm working on a backwards-compatible shorthand version for the patches to make it less tedious to write the byte arrays.

It didn't work for me, the output file is wrong, I don't know what it did but it didn't replace the string in nickel, I used the 64bit Win file.

Also command prompt close after the process is done, can you keep it open.
I used bat file with /k.

[sherman]

I just had a thought...

How are you going to handle patches that have multiple alternatives? For example "Un-Force user font-family in KePubs", where one can select one of three options?

Are you going to use the current method of commenting out the undesired options? Or is there a better way?

[jackie_w]

Originally Posted by geek1011

I'll probably add a config instruction where users can set variables and they can be substituted into any other instruction. What do all of you think of that?

I'm not exactly certain what you're proposing here but I'll just say that if I want

- CSS stream X to be set to font-family:Georgia

I may also want, for entirely different reasons,

- CSS stream Y (which controls a different part of the GUI) to be set to font-family:Avenir

[geek1011]

Originally Posted by oren64

It didn't work for me, the output file is wrong, I don't know what it did but it didn't replace the string in nickel, I used the 64bit Win file.

Also command prompt close after the process is done, can you keep it open.
I used bat file with /k.

Can you also upload nickel?

It seems like it should have worked (it worked for me):

  reading entry
    entry: ./usr/local/Kobo/nickel - size:4787988, mode:509
    checking if entry needs patching
    entry needs patching
    checking type before patching - typeflag: 48
    reading entry contents
    loading patch file: nickel.yaml
        loading patch file
        parsing patch file
        validating patch file
          ic:1
          ic:1
          ic:1
          ic:1
          ec:1, e:true, pgc:0, pg:, dc:1
          enabledPatchGroups:map[]
    applying patch file
        validating patch file
          ic:1
          ic:1
          ic:1
          ic:1
          ec:1, e:true, pgc:0, pg:, dc:1
          enabledPatchGroups:map[]
        looping over patches
          ResetBaseAddress()
          Enabled: true
        applying patch `Disable Reading Footer`
        looping over instructions
          skipping non-instruction Enabled(), PatchGroup() or Description()
          skipping non-instruction Enabled(), PatchGroup() or Description()
          FindReplaceString("ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n", "ReadingFooter {min-height:15px;max-height:15px;}\n")
            FindBaseAddressString("ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n")
            ReplaceString(0, "ReadingFooter {\n  qproperty-footerMargin: 105;\n}\n", "ReadingFooter {min-height:15px;max-height:15px;}\n")
          FindReplaceString("#caption[qApp_deviceIsTrilogy=true] {\n  font-size: 19px;\n}\n", "#caption {font-size:1px;color:transparent;}\n\x00")
            FindBaseAddressString("#caption[qApp_deviceIsTrilogy=true] {\n  font-size: 19px;\n}\n")
            ReplaceString(0, "#caption[qApp_deviceIsTrilogy=true] {\n  font-size: 19px;\n}\n", "#caption {font-size:1px;color:transparent;}\n\x00")
    copying new header to output tar - size:4787988, mode:509
    writing patched binary to output

[geek1011]

I've finished porting all the libnickel patches. Here they are (all of them are enabled by default for testing): https://github.com/geek1011/kobopatc....so.1.0.0.yaml

The SHA256 checksum of libnickel should be: 43c25f16d596076d1926822d6ce6848dc298b81dbf8b3c8071 7325d3df8cc999

Please test it and tell me if it does not match.

[geek1011]

I've released the kobopatch v0.5.0 beta. The instructions for testing it out are in the GitHub release: https://github.com/geek1011/kobopatc...ses/tag/v0.5.0

This version now includes all the patches ported from the old format. It also includes a new shorthand version for the ReplaceBytes instruction and a new FindBaseAddressHex instruction. I've tested this with all the patches and compared the output.

I would like feedback and for people to test this out (just see if it patches correctly). You don't need to install it to your kobo. Just try it out (try disabling/enabling/changing some of the patches), and upload the log, your patch files, and the generated KoboRoot.tgz. You can also check the checksums against the same settings from the old patch32lsb if you know how.

If this works well, I may actually be almost ready to release it as stable.

I have also decided to add support (in the next version) for the old patch format and even possibly calling a external patcher for easier migration.