Adding downloadable filetypes to browserd?

[kaito]

I would like to add allowable file extensions and mime types to the browserd file on my Kindle Keyboard v3.4 so I can download .azw3 and .pdf (and other) filetypes directly from Calibre web server.

My /usr/bin/browserd is 767060 bytes.

I am aware of the Free Download Hack but it seems like so much could go wrong and I'd rather not add background curl downloading if I don't have to.
https://www.mobileread.com/forums/sho...d.php?t=121008

After reading these forums it is my understanding that browserd has a hard coded list of file extensions and mime types that are allowed to be downloaded (by matching strings, and wildcards are not allowed). I would like to add to this list.

Calibre has the .azw3 has mime type:
application/x-mobi8-ebook
(I don't want to change it, I'd rather just modify my kindle ones and have it work with any Calibre server)

While using a hex editor I noticed that this older browserd was modified to allow .epub and .pdf files and it looks like they were just inserted in there?

https://code.google.com/p/gsnippet/s.../data/browserd

A8B10:  00 00 00 00-44 6F 77 6E-6C 6F 61 64-69 6E 67 2C      Downloading,
A8B20:  49 64 3D 25-70 2C 20 66-69 6C 65 3D-25 73 00 00  Id=%p, file=%s  
A8B30:  2E 61 7A 77-00 00 00 00-2E 61 7A 77-31 00 00 00  .azw    .azw1   
A8B40:  2E 70 72 63-00 00 00 00-2E 70 64 66-00 00 00 00  .prc    .pdf    
A8B50:  2E 65 70 75-62 00 00 00-2E 6D 6F 62-69 00 00 00  .epub   .mobi   
A8B60:  4C 4F 41 44-20 44 45 43-4C 49 4E 45-44 3A 20 42  LOAD DECLINED: B

But the number of 00's (nulls) between each extension seems variable. How is that determined? What is the right way to add more extensions and mime types? Doesn't inserting things into a binary harm other offsets?

Here is my unmodified binary (at a different location):

AAD40:  00 00 00 00-44 6F 77 6E-6C 6F 61 64-20 72 65 71      Download req
AAD50:  75 65 73 74-65 64 2C 20-64 6F 77 6E-6C 6F 61 64  uested, download
AAD60:  49 64 3D 25-70 2C 20 66-69 6C 65 3D-25 73 00 00  Id=%p, file=%s  
AAD70:  2E 61 7A 77-00 00 00 00-2E 61 7A 77-31 00 00 00  .azw    .azw1   
AAD80:  2E 70 72 63-00 00 00 00-2E 6D 6F 62-69 00 00 00  .prc    .mobi   
AAD90:  4C 4F 41 44-20 44 45 43-4C 49 4E 45-44 3A 20 42  LOAD DECLINED: B

AA540:  64 65 64 00-73 74 61 74-75 73 00 00-74 65 78 74  ded status  text
AA550:  2F 78 2D 70-72 63 00 00-61 70 70 6C-69 63 61 74  /x-prc  applicat
AA560:  69 6F 6E 2F-78 2D 6D 6F-62 69 70 6F-63 6B 65 74  ion/x-mobipocket
AA570:  2D 65 62 6F-6F 6B 00 00-61 70 70 6C-69 63 61 74  -ebook  applicat
AA580:  69 6F 6E 2F-6F 63 74 65-74 2D 73 74-72 65 61 6D  ion/octet-stream
AA590:  00 00 00 00-74 65 78 74-2F 70 6C 61-69 6E 00 00      text/plain  
AA5A0:  2E 74 78 74-00 00 00 00-44 65 66 61-75 6C 74 69  .txt    Defaulti
AA5B0:  6E 67 20 4D-49 4D 45 20-70 6F 6C 69-63 79 20 66  ng MIME policy f

[knc1]

Originally Posted by kaito

A8B10:  00 00 00 00-44 6F 77 6E-6C 6F 61 64-69 6E 67 2C      Downloading,
A8B20:  49 64 3D 25-70 2C 20 66-69 6C 65 3D-25 73 00 00  Id=%p, file=%s  
A8B30:  2E 61 7A 77-00 00 00 00-2E 61 7A 77-31 00 00 00  .azw    .azw1   
A8B40:  2E 70 72 63-00 00 00 00-2E 70 64 66-00 00 00 00  .prc    .pdf    
A8B50:  2E 65 70 75-62 00 00 00-2E 6D 6F 62-69 00 00 00  .epub   .mobi   
A8B60:  4C 4F 41 44-20 44 45 43-4C 49 4E 45-44 3A 20 42  LOAD DECLINED: B

But the number of 00's (nulls) between each extension seems variable. How is that determined? What is the right way to add more extensions and mime types? Doesn't inserting things into a binary harm other offsets?

Take another look at that file layout: Character + nulls == 8
(I.E: Fixed length array of strings - at least in the above example.)

[kaito]

Originally Posted by knc1

Take another look at that file layout: Character + nulls == 8
(I.E: Fixed length array of strings - at least in the above example.)

So an unlimited number of fixed length items can be added? Doesn't the array size need to be updated somewhere?

What about the mime types? Those don't look like fixed length.

Is it really all just guess work?

[knc1]

Originally Posted by kaito

So an unlimited number of fixed length items can be added? Doesn't the array size need to be updated somewhere?

What about the mime types? Those don't look like fixed length.

Is it really all just guess work?

I meant exactly what I posted in answer to how to determine the number of nulls in the specific example I quoted.

- - - - -

If the snippets you are posting are from a file similar to the browserd that you linked to - -
Then the first thing to look at is the first few bytes of the file.

Once you do that, then no guessing in this case is required.
(It is a very well defined file format, with many tools available to display the details of its content.)

[user743]

the last 16 charactars at the end of every line correspond to the 16 character sets in the middle, based on the last two characters of each Unicode character.

[kaito]

Originally Posted by knc1

I meant exactly what I posted in answer to how to determine the number of nulls in the specific example I quoted.

- - - - -

If the snippets you are posting are from a file similar to the browserd that you linked to - -
Then the first thing to look at is the first few bytes of the file.

Once you do that, then no guessing in this case is required.
(It is a very well defined file format, with many tools available to display the details of its content.)

browserd: ELF 32-bit LSB executable, ARM, version 1 (SYSV), for GNU/Linux 2.6.14, dynamically linked (uses shared libs), stripped

browserd: file format elf32-little
objdump: Can't disassemble for architecture UNKNOWN!


Sorry, I don't know how to read this file format. An example of a tool for windows x86 or linux x86 that would let me insert items into these two areas without screwing up the format would be appreciated. A generic hex editor doesn't respect its format.

[knc1]

Originally Posted by kaito

browserd: ELF 32-bit LSB executable,
ARM,
^ ^ ^ ^ ^
version 1 (SYSV), for GNU/Linux 2.6.14, dynamically linked (uses shared libs), stripped

browserd: file format elf32-little
objdump: Can't disassemble for architecture UNKNOWN!
^ ^ ^ ^ ^

So use the objdump (and/or readelf) from your cross-toolchain.
The command you gave above probably executed an "Intel arch only" version of objdump installed on the machine you where using.

[kaito]

Unfortunately I don't think I have the time to learn all about assembly for this task; the output of objdump -D isn't something I understand. I was hoping for a utility that would make the editing easy or maybe pointers from someone else that has already gone through this.

[knc1]

Originally Posted by kaito

Unfortunately I don't think I have the time to learn all about assembly for this task; the output of objdump -D isn't something I understand.
I was hoping for a utility that would make the editing easy
or maybe pointers from someone else that has already gone through this.

objcopy is your entry point to modifying a binary.
or
just generating a modified one from the sources.

Most likely, combine the two methods (the original binary + just the binary modifications):

Spoiler:

core2quad ~ $  /opt/arm-2006q3/bin/arm-none-linux-gnueabi-objcopy
Usage: /opt/arm-2006q3/bin/arm-none-linux-gnueabi-objcopy [option(s)] in-file [out-file]
 Copies a binary file, possibly transforming it in the process
 The options are:
  -I --input-target <bfdname>      Assume input file is in format <bfdname>
  -O --output-target <bfdname>     Create an output file in format <bfdname>
  -B --binary-architecture <arch>  Set arch of output file, when input is binary
  -F --target <bfdname>            Set both input and output format to <bfdname>
     --debugging                   Convert debugging information, if possible
  -p --preserve-dates              Copy modified/access timestamps to the output
  -j --only-section <name>         Only copy section <name> into the output
     --add-gnu-debuglink=<file>    Add section .gnu_debuglink linking to <file>
  -R --remove-section <name>       Remove section <name> from the output
  -S --strip-all                   Remove all symbol and relocation information
  -g --strip-debug                 Remove all debugging symbols & sections
     --strip-unneeded              Remove all symbols not needed by relocations
  -N --strip-symbol <name>         Do not copy symbol <name>
     --strip-unneeded-symbol <name>
                                   Do not copy symbol <name> unless needed by
                                     relocations
     --only-keep-debug             Strip everything but the debug information
  -K --keep-symbol <name>          Do not strip symbol <name>
     --keep-file-symbols           Do not strip file symbol(s)
     --localize-hidden             Turn all ELF hidden symbols into locals
  -L --localize-symbol <name>      Force symbol <name> to be marked as a local
     --globalize-symbol <name>     Force symbol <name> to be marked as a global
  -G --keep-global-symbol <name>   Localize all symbols except <name>
  -W --weaken-symbol <name>        Force symbol <name> to be marked as a weak
     --weaken                      Force all global symbols to be marked as weak
  -w --wildcard                    Permit wildcard in symbol comparison
  -x --discard-all                 Remove all non-global symbols
  -X --discard-locals              Remove any compiler-generated symbols
  -i --interleave <number>         Only copy one out of every <number> bytes
  -b --byte <num>                  Select byte <num> in every interleaved block
     --gap-fill <val>              Fill gaps between sections with <val>
     --pad-to <addr>               Pad the last section up to address <addr>
     --set-start <addr>            Set the start address to <addr>
    {--change-start|--adjust-start} <incr>
                                   Add <incr> to the start address
    {--change-addresses|--adjust-vma} <incr>
                                   Add <incr> to LMA, VMA and start addresses
    {--change-section-address|--adjust-section-vma} <name>{=|+|-}<val>
                                   Change LMA and VMA of section <name> by <val>
     --change-section-lma <name>{=|+|-}<val>
                                   Change the LMA of section <name> by <val>
     --change-section-vma <name>{=|+|-}<val>
                                   Change the VMA of section <name> by <val>
    {--[no-]change-warnings|--[no-]adjust-warnings}
                                   Warn if a named section does not exist
     --set-section-flags <name>=<flags>
                                   Set section <name>'s properties to <flags>
     --add-section <name>=<file>   Add section <name> found in <file> to output
     --rename-section <old>=<new>[,<flags>] Rename section <old> to <new>
     --change-leading-char         Force output format's leading character style
     --remove-leading-char         Remove leading character from global symbols
     --redefine-sym <old>=<new>    Redefine symbol name <old> to <new>
     --redefine-syms <file>        --redefine-sym for all symbol pairs 
                                     listed in <file>
     --srec-len <number>           Restrict the length of generated Srecords
     --srec-forceS3                Restrict the type of generated Srecords to S3
     --strip-symbols <file>        -N for all symbols listed in <file>
     --strip-unneeded-symbols <file>
                                   --strip-unneeded-symbol for all symbols listed
                                     in <file>
     --keep-symbols <file>         -K for all symbols listed in <file>
     --localize-symbols <file>     -L for all symbols listed in <file>
     --globalize-symbols <file>    --globalize-symbol for all in <file>
     --keep-global-symbols <file>  -G for all symbols listed in <file>
     --weaken-symbols <file>       -W for all symbols listed in <file>
     --alt-machine-code <index>    Use the target's <index>'th alternative machine
     --writable-text               Mark the output text as writable
     --readonly-text               Make the output text write protected
     --pure                        Mark the output file as demand paged
     --impure                      Mark the output file as impure
     --prefix-symbols <prefix>     Add <prefix> to start of every symbol name
     --prefix-sections <prefix>    Add <prefix> to start of every section name
     --prefix-alloc-sections <prefix>
                                   Add <prefix> to start of every allocatable
                                     section name
  -v --verbose                     List all object files modified
  @<file>                          Read options from <file>
  -V --version                     Display this program's version number
  -h --help                        Display this output
     --info                        List object formats & architectures supported
/opt/arm-2006q3/bin/arm-none-linux-gnueabi-objcopy: supported targets: elf32-littlearm elf32-bigarm elf32-little elf32-big srec symbolsrec tekhex binary ihex

Reading the ARM instructions that you just dump'd will answer your questions about
"Where / How do I change the limit on the array length?"
"What is the storage format for mime types?"
and just about any other question you might have about how it works / how to change it.

Sometimes changing the behavior of a binary does include the WAFG (Wild Ass F...ing Guess) protocol you mentioned - but only rarely.

C.S. -> The "S" stands for: Science - - - There is a method to all this madness here.

- - - - -

Hint:
Give up on the built-in browser - just install one of our alternate browser packages.
(If we have a build for the K3 that is, I don't recall off-hand if we do.)

[kaito]

Originally Posted by knc1

Hint:
Give up on the built-in browser - just install one of our alternate browser packages.
(If we have a build for the K3 that is, I don't recall off-hand if we do.)

I wish there was one, but I looked and didn't find any.

Originally Posted by knc1

objcopy is your entry point to modifying a binary.
or
just generating a modified one from the sources.

Most likely, combine the two methods (the original binary + just the binary modifications):


Reading the ARM instructions that you just dump'd will answer your questions about
"Where / How do I change the limit on the array length?"
"What is the storage format for mime types?"
and just about any other question you might have about how it works / how to change it.

Sometimes changing the behavior of a binary does include the WAFG (Wild Ass F...ing Guess) protocol you mentioned - but only rarely.

C.S. -> The "S" stands for: Science - - - There is a method to all this madness here.

I'm sure everything has an answer, but since I have a limited life span I must prioritize my time. For the task at hand (simply downloading azw3 and pdf files on my old k3) I can't invest a large amount of time learning how to use objcopy. I was hoping for a smart gui editor that would read the file, understand its format and let me make modifications without harm -- or hand holding on how to do it in under ~20 minutes (maybe the internet has this specialized knowledge somewhere, but google hasn't been very helpful to me).

[user743]

deleted

[knc1]

Originally Posted by kaito

I'm sure everything has an answer, but since I have a limited life span I must prioritize my time. For the task at hand (simply downloading azw3 and pdf files on my old k3) I can't invest a large amount of time learning how to use objcopy. I was hoping for a smart gui editor that would read the file, understand its format and let me make modifications without harm -- or hand holding on how to do it in under ~20 minutes (maybe the internet has this specialized knowledge somewhere, but google hasn't been very helpful to me).

Well, that is a different set of circumstances than your original post:

Originally Posted by kaito

I would like to add allowable file extensions and mime types to the browserd file on my Kindle Keyboard v3.4 so I can download .azw3 and .pdf (and other) filetypes directly from Calibre web server.

Ah, and you are concerned about a limited life span?

So instead, you want me (very near the big 70 myself) to work out the solution and hand it to you?
What about my limited life span?

I don't want this change, you want it, you invest your life span in learning how to do it.

Or just use the download hack linked to in the above post - the one you refused to use earlier.

[kaito]

Originally Posted by knc1

Well, that is a different set of circumstances than your original post:

Ah, and you are concerned about a limited life span?

So instead, you want me (very near the big 70 myself) to work out the solution and hand it to you?
What about my limited life span?

I don't want this change, you want it, you invest your life span in learning how to do it.

Or just use the download hack linked to in the above post - the one you refused to use earlier.

I don't want you to invest anything that you don't want to invest. It was a question to all, not only directed at you.

I am aware of that hack (as I said in my original post), but why have a forum if everyone should just go and figure things out on their own? Why have communication or language at all? Silly social human beings.

I am still interested in hearing from anyone else that has a helpful answer that would accomplish my original goal of editing that file. It isn't a life or death issue, just something I wanted to do.

[knc1]

Originally Posted by kaito

- - - -
I am aware of the Free Download Hack but it seems like so much could go wrong and I'd rather not add background curl downloading if I don't have to.
https://www.mobileread.com/forums/sho...d.php?t=121008
- - - -

You have to.
Your other requirements have eliminated any choice.

- - - -

The true mysteries of life: