iLiad iLiad 3rd Party programs standardization

[Adam B.]

Given that an unbricking option is on the horizon, and many new developers will probably start to emerge, I think now is a good time to talk about standardizations in our applications. Nick (smoogle) and I talked a bit about this, and I’d like for everyone to come to a consensus about the best way to approach this.

Packaging and Installation.

Installing apps should be as easy as possible for the average user. I like the way things are done now, a single folder containing everything the user needs to run a program. Copy it to the iLiad, select from the contentlister, and you’re good to go. I also like the way that mtas has these setup in the main menu of the contentlister as well (https://www.mobileread.com/forums/sho...18&postcount=6). However, that should be up to the user.

I’d like for a program to be able to be run from anywhere the user chooses to put it. This can be complicated depending on the program. Nick has made some changes to FBReader that will allow it to run from anywhere (source attached). However, not all programs may be able to be modified to do this without extensive code changes. The more we modify a program, the more difficult it is to update it when a new version is released. What if we were to compile a program with a specific working directory. Let’s say for example /home/root/Apps/programname. This way when the program is run, the start script can check if that directory exits. If it does not, make a symbolic link from there, to the directory the script is actually running from and then execute the program. I don’t know if it is also possible within a script to check as to where the symbolic link is pointing to. This way, if a user moves the folder, after the initial run, the program will still work.

I don’t know if /home/root is the best place to put it, since it will be cleared after every software upgrade. Maybe /mnt/settings would be better. But if the script checks for it’s existence every time, it may not be an issue.

Also, after some thought, the startup script should set the HOME environment variable to the current directory. I didn’t think this would matter at first, but as filling up the root directory can be a real possibility, storing everything on the memory card is a good idea. Besides, that will also survive a software flash.

I don't know anything about ipkg. If we can make an easy way for the user to install it, then I'm all for it. But to select it from the contentlister, changes will need to be made to er_registry.

Shared Libraries

Every program that I’ve ported requires shared libraries. Normally, I include these in a lib directory within the program’s folders. However, as more and more programs are ported, the more space is being wasted by duplicate files. Perhaps in the initial launching script, we can copy the libraries needed with the program to a shared library directory somewhere on the memory card. It should check to see if it is launching from /media/card or /media/cf or /mnt/free. Then, create a folder (if it does not exist) /mnt/wharever/_sharedlibs. From there a script as antartica describes (https://www.mobileread.com/forums/sho...4&postcount=17) would work well.

Also, when a conflicting version is found, it can retain it’s local copy. In the script, we can set the LD_LIBRARY_PATH=/`pwd`/lib:/media/cf/_sharedlibs:/media/mmc/_sharedlibs:/mnt/free/_sharedlibs. This way, I believe it will use the first shared library it finds within it’s own directory.

I had some more ideas, but my mind just went blank….I’ll post more as it comes to me.

What I'd like to do, is make a list of guidelines and files (startup script, manifest template, etc) that should be included when packaging a program for the iLiad. This way users won't be confused with different releases from different developers.

[smoogle]

Some further comments;

It is possible (and I intend to), make the new code for relocation to be done as a .patch file; allowing us to apply it to whichever version we wish. It is also to our advantage, that code like that in fbreader/etc is not often (if ever) changed across revisions.

I agree entirely with the LD_LIBRARY_PATH stuff; but another neat thing I thought of-

We could make a program to run on startup, and verify er_registry.txt; a common problem I understand is if the file is not valid, the iLiad won't start the contentlister? So i'm going to work on a program that checks if the file is valid; if the file isn't valid for some reason, then it can use a safe backup.

Also, fyi you can find out where symbolic links point to, but I don't think it's too imporant? We can just have the script update the location each time it's run (not like creating a symbolic link is a time-consuming process anyway).

[Henry Loenwind]

I like the idea of using ipkg to install third party programs better than using directories for the user to put onto an SD/CF card. I see three main problems with the directories approach:

(1) The user must unzip (untar, unrar, ...) an archive and then put a directory with a bunch of files on a memory card. For the average computer user this is (according to my experience) just nearly impossible. And be realistic: On the long run, the iLiad should be usable for people who don't have a clue about computers.

(2) Filename case will be shredded by this process, as the archive will be unzipped on a Windows PC in most cases. Hey, that bit me just yesterday when I tried to install Perl.

(3) The custom startscripts need to be somewhat complex. There will be dozens of versions and every programmer will roll his own speciality. How long until a "rm" (I read "recreate symlink earlier in this thread!) will misfire after the script has been released? (And I can remember one developer's bricked iLiad...)

Using ipkg and a predefined directory and symlink structure has the advantage that the user has to run a "setup script" (a single .sh file) once, and then could put .ipkg onto the iLiad to be installed. First we have to deleiver a shell script with every ipkg (containing just the ipkg call), but as soon as iRex allows for save content lister extension, .ipkg files could be made directly clickable by the setup script.

However, we should define a directory structure first, so we create us another big mixed-up disorder...

Allow me to start:

---

(1) Prior to installing the iLiad must be prepared:

mkdir /mnt/cf/opt && ln -s /mnt/cf/opt /opt
or
mkdir /mnt/card/opt && ln -s /mnt/card/opt /opt

mkdir /opt/lib
mkdir /opt/bin
mkdir /opt/etc
mkdir /opt/var
mkdir /opt/programs

One of these, as fs allows:
ln -s /opt/programs /mnt/cf/programs
ln -s /opt/programs /mnt/card/programs

This is what should go into the setup script. With some simple shell programming it can also check if there is a CF or a SD card present and create the target directory on the correct media. With the symlink as /opt, the internal filesystems of the iLiad will not be touched except for one single symlink. So the setup script will not be able to brick it.

(2) Installation packages should follow these rules:

• All files shall be installed into /opt, no other part of the filesystem may be touched.
• If the program is to be started directly (content lister sh-extension), it shall be installed into /opt/programs. It shall have its MANIFEST in its top-level folder (e.g. /opt/programs/fbreader/MANIFEST.xml).
• Common shared libraries (.so) go into /opt/lib.
• Configuration settings for into /opt/etc. Using a subfolder is recommended if having more than one file, e.g. /opt/etc/fbreader/.
• Programs that are to be started from the command line, or that are added as content handler to the content lister, go into /opt/bin.

And as everything is installed into /opt, again the internal filesystems will not be touched (except for ipkg's database). No chance of bricking, if we add a check for /opt being a symlink into the ipkg-wrapper script. (and/or the preinstall script of the ipkg)

---

Is this a reasonable start? What do you developers think about it?

(I could implement the needed scripts, but I want this issue being discussed first, and not rush into creating facts.)

[Adam B.]

The only problem I see with this approach, is that opt and programs would be visable in the contentlister to the user when browsing the memory card. I think that keeping the card as clean as possible when viewing on the iLiad would be ideal. I think that an underscore before the folder name will hide it, if not I would assume that a period would.

Also, I've never created an ipkg before. How involved is it? How smart is ipkg (as far as libraries and dependencies go)?

[smoogle]

Hello, i'm just going to respectfully disagree with your preferrence of ipkg; here are my replies initially to your points;

1) The single distinction is the use of a single file (ipkg) versus an archive which must be extracted. I don't think this is a significant hurdle for users (reasons follow).

2) Filename case should be fine if the archive is created correctly- I do not see how your error could have occured (at all).

3) The motivation is to create a single standardised script, that all programmers use. By following a per-subfolder fixed structure the script could operate in the same fashion for all programs.

Finally, the biggest point I have, is we could simply ipkg the above folder and use your ipkg-ing script (minus the need for any setup), or alternatively, as you suggest for the .ipkg extension we could support the .tar.gz extension and allow you to directly extract from within the content lister.

I do like your suggestion of a fixed filesystem at /opt, but it isn't that different form Adam B's suggestion, simply /opt instead of /home/root/Apps, personally I prefer /opt. However, I still prefer a distributed approach with symlinks rather than placing the opt directory on the card.

The final problem is to do with the 'unbrickability' of it; for some programs the reason the root filesystem must be modified, is to support the contentlister, or to add new fonts/etc (as in AbiWord); these would both not be 'fixed' by the use of an /opt directory, not saying the alternative works - but just pointing out a limitation.


The setup script I would propose would work like this:
[0) Extract the archive in-place]
1) Create /opt/programs if it does not exist - *on* the root filesystem
2) Symlink /opt/programs/fbreader to the location of this script.
3) Create a _libs directory on whatever medium is used (be in CF/MMC/root) if it does not exist.
4) Optionally, move any required libraries to the _libs directory
5) Delete any libraries from the package that are contained within _libs (if you do step #4, this implies deleting all libraries).
6) Run the program.

The other directories /opt/etc could be created as well if desired, but they would all be as symlinks to the program's location of the files, rather than copies of the files.

Now, advantages:

1) No setup. You copy the file, run the script, and it makes sure it'll work with wherever you've placed it.
2) Root filesystem changes are just symlinks elsewhere, as with yours.
3) No requirement for a fixed directory structure overall.
4) If you want, the packaging method could be ipkg/tar/whatever, as long as the script can expand it.
5) Removal is simply deleting the program folder.
6) To move the program you simply put it somewhere new and run it.

the important point to make, is the visible disk structure by both methods is the same, the key difference is this way we have relocatable folders that can be placed on whatever storage you want - you don't need to have any per-CF card setup for it to work,

I just sorta like the flexibility; and combined with a util to test whether symlinks still point to valid locations, it could even prompt you "Sorry, I couldn't find fbreader, please insert the appropriate memory card."

[narve]

Originally Posted by smoogle

I just sorta like the flexibility; and combined with a util to test whether symlinks still point to valid locations, it could even prompt you "Sorry, I couldn't find fbreader, please insert the appropriate memory card."

Sorry for being off-topic, but could a developer make an extremely simple app that displays a message box? Syntax "showdialog "my message here"? Then the scripts could use error codes to show an error dialog whenever an application fails to launch or something like that. Today, if a community application fails we have to launch it via terminal windows to see what the error is, which I guess excludes many users from giving proper error reports...

[smoogle]

Originally Posted by narve

Sorry for being off-topic, but could a developer make an extremely simple app that displays a message box? Syntax "showdialog "my message here"? Then the scripts could use error codes to show an error dialog whenever an application fails to launch or something like that. Today, if a community application fails we have to launch it via terminal windows to see what the error is, which I guess excludes many users from giving proper error reports...

Done, but not compiled, takes a single commandline parameter and pops up a centered message box. Should work... It's just a simple GTK application. I'll test it later;

compile using something like the following:
gcc -o mbox mbox.c `pkg-config gtk+-2.0 --cflags --libs`

[pdam]

I'm not a developer ... but - I'm not sure I agree with the share LIB idea totally, if programs are going to run off of memory cards. It's good to have the complete app bundled for your average user with everything needed in the bundle (ala Portable Apps for windows). Space on the Iliad is an issue, very rarely will it be an issue on an SD card. If everything is in the same directory and you simply put this on an SD card and run a script - it makes life easy for your average user ...

[tororebelde]

It's an interesting discussion here, for me. I agree most of the ideas contributed here:

I do like your suggestion of a fixed filesystem at /opt, but it isn't that different form Adam B's suggestion, simply /opt instead of /home/root/Apps, personally I prefer /opt. However, I still prefer a distributed approach with symlinks rather than placing the opt directory on the card.

I agree.

The final problem is to do with the 'unbrickability' of it; for some programs the reason the root filesystem must be modified, is to support the contentlister, or to add new fonts/etc (as in AbiWord); these would both not be 'fixed' by the use of an /opt directory, not saying the alternative works - but just pointing out a limitation.

When a unbrickable solution given, I'm sure this can be solved. If contentlister becames more flexible to allow us add our stuff, can help but this is a iRex alternative and no way until they do the next move and show us their solution to unbrickability.

The setup script I would propose would work like this:
[0) Extract the archive in-place]
1) Create /opt/programs if it does not exist - *on* the root filesystem
2) Symlink /opt/programs/fbreader to the location of this script.
3) Create a _libs directory on whatever medium is used (be in CF/MMC/root) if it does not exist.
4) Optionally, move any required libraries to the _libs directory
5) Delete any libraries from the package that are contained within _libs (if you do step #4, this implies deleting all libraries).
6) Run the program.

This is good to me, but the fourth point I'm not sure how to deal with it. It has been discussed to use a common /opt/lib directory, and is the better for me, but since this will fast decrease free internal memory, a _libs on cards looks good for me. But don't understand your solution to avoid duplicated libs. Maybe I'm lost on translation?

Anyway, I will be aware to what you guys say on this thread and I will help in all I can!

[smoogle]

By 'optionally', I meant present user with a message box perhaps, or just default appropriately; as for versions we'd simply need to ensure we give different versions distinct names.

As for the _libs on the card suggestion, my idea was you keep the libraries stored in the same location as the program. For instance running it on a CF card, leaves the libs there- but they can be shared with other apps on the same card.

As for lib duplication, sadly it adds up quite quickly, when some libraries like libX11 consume 4MB each...

[mtas]

Originally Posted by smoogle

I do like your suggestion of a fixed filesystem at /opt, but it isn't that different form Adam B's suggestion, simply /opt instead of /home/root/Apps, personally I prefer /opt. However, I still prefer a distributed approach with symlinks rather than placing the opt directory on the card.

I just sorta like the flexibility; and combined with a util to test whether symlinks still point to valid locations, it could even prompt you "Sorry, I couldn't find fbreader, please insert the appropriate memory card."

A few observations.

I would not recommend using /opt or /home/root/Apps. Both of them resides on the root file system. Also, /opt is already used by the iLiad.

Instead, I would suggest something like this:
Use an initial installation script that creates the directories _additional_progs and _additional_progs/_local in the root of the file system it is run from. I.e. if it's run from a CF card it should create /media/cf/_additional_progs and /media/cf/_additional_progs/_local. Then let the script create a symlink from /usr/local to /media/cf/_additional_progs/_local.

When this is done then we can use _additional_progs for start-up scripts for programs that are launched from the content lister and install the software itself in the _additional_progs/_local directory.

By doing it in this way we don't have to worry about where the installed files are located since they will be referenced via the /usr/local symlink. Also, if the media card is removed we won't have any broken references to the extra applications installed.

The creation of a standard installation method is really needed but it's also a bit more complex than it seems at the first glance. We need not only to define how to install and execute the programs. We need to create the whole infrastructure needed, the same install/de-install scripts should be used and "global" environment variables like HOME and LD_LIBRARY_PATH should be defined.

The infrastructure issues is IMHO the biggest reason for using ipkg, there we already have the basic tools needed.

[smoogle]

Perhaps if /usr/local was a normal folder, containing symlinks?

The problem is more related to support you had multiple CF cards, or wanted to run apps from the root/CF/MMC all at once- linking per-app allows you to achieve this.

Also, linking to stuff that has been removed isn't nessecarily a bad thing (for instance we can prompt media re-insertion); the main reason I was thinking to have a single folder that lists all the application startup scripts, is then we can standardise tools to edit er_registry and so on by examining this list.

I'm a big fan as you may have guessed of eliminating the concept of install/uninstall

[mtas]

Originally Posted by smoogle

Perhaps if /usr/local was a normal folder, containing symlinks?

The problem is more related to support you had multiple CF cards, or wanted to run apps from the root/CF/MMC all at once- linking per-app allows you to achieve this.

Also, linking to stuff that has been removed isn't nessecarily a bad thing (for instance we can prompt media re-insertion); the main reason I was thinking to have a single folder that lists all the application startup scripts, is then we can standardise tools to edit er_registry and so on by examining this list.

I'm a big fan as you may have guessed of eliminating the concept of install/uninstall

Yes I can see that and I'm all for it. I'm creating a first version of my suggestion right now so in a day or two you'll be able to testdrive it. . .

[Adam B.]

Originally Posted by tororebelde

This is good to me, but the fourth point I'm not sure how to deal with it. It has been discussed to use a common /opt/lib directory, and is the better for me, but since this will fast decrease free internal memory, a _libs on cards looks good for me. But don't understand your solution to avoid duplicated libs. Maybe I'm lost on translation?

The actual libs won't be stored on the internal memory, but linked to whatever memory card you use.


I heard someone say that /opt is already in use on the iLiad, how about /var/opt? From there we can implement something similar to what Henry mentioned, but instead link it to /media/???/_opt. In there, we can have share, bin, lib, etc.

Using ipkg isn't a bad idea. We wouldn't even need to make any changes to the contentlister to run it (thus preventing problems with a user running too large of an ipkg file). We can distribute the ipkg along with an installation script. It'll create the necessary directories and symlinks, remove the installation file, and change itself from an installer to the program launcher. Should be simple to do and easy for the user to understand.

I don't think that zipping the needed ipkg, script, manifest, and icon would cause much of an issue. XP has zip functionality built in. A user can easily open a zip as a folder and drag and drop into the memory card. Either that, or go through xp horrible zip wizard...

[narve]

Originally Posted by Adam B.

I don't think that zipping the needed ipkg, script, manifest, and icon would cause much of an issue. XP has zip functionality built in. A user can easily open a zip as a folder and drag and drop into the memory card. Either that, or go through xp horrible zip wizard...

I've had bad experiences with this one... my iLiad is getting quite flaky when i've copied large amounts of files using USB cable. Could be my memory card, though.

Anyways, I prefer downloading files using wget or scp and unzipping on the iLiad. For some reason, zip doesn't work on the iLiad so tar.gz releases would be appreciated. If the zip-file is created by a build script, adding tar.gz support should be easy.

And for XP-users: The built-in zip functionality is extremely crappy. 7-zip is a free/open alternative which is a lot faster and has lots more functionality as well as a vastly superior interface (i.e. no wizard!).