Linux, Mac OS X, and Windows Application Packaging

[PeterT]

@KevinH and @DiapDealer (and others):

See this post over in the Windows 10 thread https://www.mobileread.com/forums/sho...postcount=1943

[KevinH]

Hi PeterT,

Originally Posted by PeterT

@KevinH and @DiapDealer (and others):
See this post over in the Windows 10 thread https://www.mobileread.com/forums/sho...postcount=1943

That is basically my exact feeling as well (except for the Windows 10 vs Mac OS X bias ;-). What few commercial packages I have seen for Linux try and either locate in /usr/local/PACKAGENAME or /opt/PACKAGENAME just to not be spread all over tarnation.

And requiring your entire OS to be recompiled just to update a specific library at times, is a sure sign of insanity. Linux really needs to move to a hybrid model were simple basic commands and shared libraries are still stored in the current Linux hierarchy, but actual applications are stored in a standard location in some consolidated form as on Windows (C:\Program Files\PACKAGE) and Mac OS X (/Applications). Then add in application containers to allow drag and drop of entire applications, code signed verification on load, restricted access to resources by application, and sandboxing and you will bring Linux kicking and screaming into the present.

I also agree the Linus is correct, the plethora of Linux distributions with their arcane rules and distribution specific changes has done more to damage Linux on the Desktop than almost anything else.

Nice post.

KevinH

[eschwartz]

Isn't /opt pretty much the exact "consolidated form" from Windows/OSX you want, minus code signing?

[KevinH]

Hi

Originally Posted by eschwartz

Isn't /opt pretty much the exact "consolidated form" from Windows/OSX you want, minus code signing?

It is the start of it.

From the Desktop these need to be easily able to run without requiring lots of PATH of LD_LIBRARY_PATH hacks. It needs to have some form of sandboxing possible. It needs to be signable with verification on application launch. It needs have a way to keep a list of rights and resources the app is allowed to use. It needs to allow private Frameworks and other internal app structures to support internal resources, it needs to appear like a single file to the user and be drag and droppable for installation easily. Something like a separate file fork would be useful to store metadata about the app and extended attributes for the file in a manner invisible to the end user and to prevent the need for central registry nonsense, etc.

I have read a few posts about application containers and their ability to run each app in its own VM sandbox that is being developed on Linux. Hopefully this would bring along many of the other desirable traits. If so, it certainly be a big step forward.

KevinH

[eschwartz]

Well, I suppose you are eagerly awaiting some of that in Windows as well.
Last I checked, "applications appearing like a single file" is quite OSX-specific. Windows uses installers instead, and linux its package managers -- package managers could install commercial applications, if anyone could ever agree on a common format.

Minor trivia -- the command-line tools for calibre are unavailable on OSX, at least without manually making symlinks deep into the heart of the Schrödinger's file of an Application, after installation, because of OSX's Application bundle model. I've gotten around that, via an OSX port of my calibre-upgrade.sh here: https://github.com/eli-schwartz/calibre-installer -- which breaks the Application-is-a-drag-and-drop-file model also.
On linux, applications go in `/usr/bin` and access their `/usr/lib`'s etc. and are easily available as a matter of course. Or install to /opt and symlink to /usr/bin, possible because of the package manager strategy.
On Windows, the MSI can add calibre's installation directory to the PATH.
Maybe there is something similar that can be done for OSX, I don't know. Apparently neither does Kovid.


Verify on application launch as opposed to signed installers?
That is orthogonal to your original complaint. As are sandboxes.

That relates to the security model, not the ability to publish an application that doesn't need recompilation for every trivial OS component update.

And some of that has been around for a while. SELinux, containerization, both are real today.
Frankly, they are only desperately urgent in a more commercial environment.


registry nonsense -- Windows, not linux?
Or do you mean that each file should be allowed to override its own mimetype and say which applications should open it, and each application should be allowed to, um, what?
Forgive me, I have no idea what OSX is doing, much less the motivation.

[KevinH]

Hi,
On MacOSX, Yes you can add any directory to the path for any user just like any other unix box (play with their bash startup scripts or system bash startup scripts). As for code verification on load, that is much more desirable than just signing the original package as it helps prevent any improperly changed code from launching. Single file drag and drop installation and distributed metadata, makes installation and package management much easier without the central failure point of a registry. This all in turn allows much more robust sandboxing.

Linux is simply behind the times in that and as Linus pointed out, the distribution specific changes are killing commercial dev for Linux.

I am hoping the new work on the Linux App Containers will fix most of these issues.

Not sure what you are arguing against here. Linus even agrees with me!

[eschwartz]

Originally Posted by KevinH

Hi,
On MacOSX, Yes you can add any directory to the path for any user just like any other unix box (play with their bash startup scripts or system bash startup scripts).

It is ironic you say that, immediately after complaining that linux requires you to mess around with LD_LIBRARY_PATH.

And it is IMHO unreasonable to expect people to maintain lengthy PATHs in their bashrc in order to access the CLI functionality of an Application bundle that has CLI functionality.
Especially because it requires investigating what to add. People have asked here a number of times how to do it, because they tried accessing the CLI tools via what they thought was the right path, but which apparently wasn't.


Anyway. This was just an example of one way in which I think Application bundles may have their weak points.

As for code verification on load, that is much more desirable than just signing the original package as it helps prevent any improperly changed code from launching.

I am not afraid of my code being changed. It is owned by root.
The kernel is also owned by root. Maybe they will change that instead.

Okay, it's good-to-have. But I am pretty sure the current lack of it is NOT what prevents the mystical Year of the Linux Desktop.

Single file drag and drop installation and distributed metadata, makes installation and package management much easier without the central failure point of a registry. This all in turn allows much more robust sandboxing.

Linux is simply behind the times in that and as Linus pointed out, the distribution specific changes are killing commercial dev for Linux.

I am hoping the new work on the Linux App Containers will fix most of these issues.

Not sure what you are arguing against here. Linus even agrees with me!

Linus certainly seems to agree on the need for /opt applications which don't require recompiling every time the OS gets a trivial update.
But I do not seem to see where he agrees with any of your other points.

Both Windows and linux don't have a drag-and-drop installation.
Both store metadata in central locations. Windows has its registry, linux uses desktop files and custom-mimetype.xml stored in central directories, owned by packages, which are cached by the MIME database and can be trivially rebuilt from scratch.
(Very pissed, by the way, that defaults are essentially random and also sometimes change at random. Windows 8 fixed that, but did linux? noooo...)
As far as I know, Linus is fine with both of those.
As far as I know, the linux package manager model is usually held up as an example of why Windows is/was massively behind the times (no centralized source for installation and updates? ) -- excepting the fragmentation of distros duplicating the same efforts, which is of course a huge problem for commercial adoption, making it equivalent to Windows.
OSX of course has had an App Store for a while. Just like linux packaging except all Applications use isolated installs (linux could, technically, do the same if that is how the distros chose to compile an application. But they won't...)


So all three of us agree on the need for some equivalent to Windows applications that bundle their own dependencies so long as you have a working kernel (and Linus doesn't break userspace).
As far as that goes.



Which is all I am saying, really. You seem to be discussing a couple different points lumped together.
And they seem to be OSX > Windows & linux
Not Windows & OSX > linux

[BetterRed]

Posts 78-85 have very little to do with Sigil, epubs or the editing thereof.

How about tacking them onto the back of that other Off Topic thread the Sigil mods recently created

BR

[eschwartz]

Sounds eminently reasonable.

[KevinH]

Originally Posted by eschwartz

It is ironic you say that, immediately after complaining that linux requires you to mess around with LD_LIBRARY_PATH.

Not really, simple command line tools should be installed in standard locations and use the PATH, both Linux and Mac OS X have that right.

And it is IMHO unreasonable to expect people to maintain lengthy PATHs in their bashrc in order to access the CLI functionality of an Application bundle that has CLI functionality.
Especially because it requires investigating what to add. People have asked here a number of times how to do it, because they tried accessing the CLI tools via what they thought was the right path, but which apparently wasn't.

It is no different from what you have to do on Linux. What's your point?

Anyway. This was just an example of one way in which I think Application bundles may have their weak points.

Application bundles are not needed for simple command line tools. That is why a hybrid system allowing the best of both words is the correct path forward in my opinion.

I am not afraid of my code being changed. It is owned by root.
The kernel is also owned by root. Maybe they will change that instead.

You will hate to know that the latest Mac OS X El Captian defaults to disabling root entirely (ie rootless) with no way to change that unless you reboot in a special way. root still exists, it is just that nothing can become root (sudo and su stops working) and a set of system level directories are locked in stone, unable to be touched in any way. If interested:

google El Captian System Integrity Protection

Linus certainly seems to agree on the need for /opt applications which don't require recompiling every time the OS gets a trivial update.
But I do not seem to see where he agrees with any of your other points.

No he agrees that distribution differences and the need to recompile are both what is killing the Linux Desktop. For the Linux Desktop to succeed, the Linux distributions must agree on how to handle all of these issues or be stuck in the past so to speak.

Both Windows and linux don't have a drag-and-drop installation.
Both store metadata in central locations. Windows has its registry, linux uses desktop files and custom-mimetype.xml stored in central directories, owned by packages, which are cached by the MIME database and can be trivially rebuilt from scratch.
(Very pissed, by the way, that defaults are essentially random and also sometimes change at random. Windows 8 fixed that, but did linux? noooo...)
As far as I know, Linus is fine with both of those.

Then he has changed his mind. Many many years ago I remember reading a rail of his against the damn registry nonsense. I will look for the link.

As far as I know, the linux package manager model is usually held up as an example of why Windows is/was massively behind the times (no centralized source for installation and updates? ) -- excepting the fragmentation of distros duplicating the same efforts, which is of course a huge problem for commercial adoption, making it equivalent to Windows.

I am not disagreeing. I think Windows could make some improvement here as well. As could Mac OS X by the way. No Desktop system is perfect, but a lot of new ideas have come forward recently and it seems like no one on the Linux side sees their value to help unify and fix the Linux Desktop.

OSX of course has had an App Store for a while. Just like linux packaging except all Applications use isolated installs (linux could, technically, do the same if that is how the distros chose to compile an application. But they won't...)

Agreed. Getting the different Linux distributions to make any consistent changes (or even decide on a standard Desktop user experience!) has been like herding cats. It simply isn't going to get done until someone like Linus pushes for it. At least, he is now seeing it as the problem it has always been.

So all three of us agree on the need for some equivalent to Windows applications that bundle their own dependencies so long as you have a working kernel (and Linus doesn't break userspace).

Yes but I am hoping for much more from Linux. They have a chance to set up something really good here if they can just focus and get something done. I am a big user of Linux but not for Desktop use. I use it to house my research data, to create software to manipulate my data and run statistical analyses. Almost all via ssh. I would love to be able use Linux on the Desktop in some sane way that did not require me to basically become my own maintainer.

Which is all I am saying, really. You seem to be discussing a couple different points lumped together.
And they seem to be OSX > Windows & linux
Not Windows & OSX > linux

Not exactly. All of these are related to the user experience and as a developer, I have to worry about and deal with all of this nonsense, as do you.

Take care,

KevinH

[chrisridd]

Would linking the Linux binaries using -R $ORIGIN/../path/to/my/libraries be a reasonable way of making the Linux binaries self-contained and still movable? Users do not need to mess with the dangerous LD_LIBRARY_PATH environment variable...

Caveat: I've only used $ORIGIN on Solaris with the Solaris linker. I don't know if the Linux linker(s) can handle it. Getting '$ORIGIN' as a string through a build system can be entertaining ;-)

[KevinH]

That is effectively how Mac OS X handles it as well.

Libraries are linked with a full path name and that name can include placeholders like @loader_path for dlopen'd libraries and @executable_path to find the libraries relative to the current locations of the main executable.

The nice thing is that there is a simple command line tool "otool" that will allow you to set the name and path to the libraries post build so you do not need to worry about passing these things along inside the build environment.

We use these tricks to make the embedded Python.framework inside of Sigil.app fully relocatable since the Sigil.app itself is full relocatable on Mac OS X.

I thought Linux had similar RPATH and name handling capabilities but I am not sure if it works in the same way.

KevinH

Originally Posted by chrisridd

Would linking the Linux binaries using -R $ORIGIN/../path/to/my/libraries be a reasonable way of making the Linux binaries self-contained and still movable? Users do not need to mess with the dangerous LD_LIBRARY_PATH environment variable...

Caveat: I've only used $ORIGIN on Solaris with the Solaris linker. I don't know if the Linux linker(s) can handle it. Getting '$ORIGIN' as a string through a build system can be entertaining ;-)

[DiapDealer]

If you're speaking specifically of Sigil, then "users" don't have to mess with the LD_LIBARY_PATH right now. Only packagers and knowledgeable tweakers might. In fact, in the current sigil, modifying LD_LIBRARY_PATH is unnecessary for a normal build/install (I leave it in to give me options for post-installalation diddling/repackaging). Cmake already takes care of modifying the rpath with the location of the necessary external libs (specified at build-time). The main purpose of the launch script right now, is for the setting of other env vars that allow the relocation of Sigil's support files (INSTALL_PREFIX/share/sigil), and for specifying alternative hunspell dictionaries, and for informing the Python plugin interpreter (system or bundled) where libhunspell is located so plugin devs can utilize our built-in ctypes interface to it.

[kovidgoyal]

Linux also supports that, and even has a tool to easily change it called chrpath. However, I find that LD_LIBRARY_PATH works great. Just a couple of things you have to keep in mind about it:

1) When running subprocesses from outside your application bundle, remember to remove the directory you added from LD_LIBRARY_PATH in the environment the subprocess gets. For instance, when calling

QDesktopServices::openURL

see the sanitize_env_vars() function int he calibre source code.

2) Instead of writing a bash script to launch your executable, use a little C program instead, its faster and effectively shuts up the whining from distro maintainers See linux/launcher.c in the calibre source code for an example.

@KevinH: the problem in OS X is with application bundles installed with the DMG installer that have both a gui and command line tools. There is no way to install command line tools into the PATH at install time, since you can run no scripts at install time. And even on first run it cannot be done because the user account running the GUI typically does not have permission to write to the directories on PATH like /usr/bin

So that means you either have to use the OS X authorization API to run a helper program as root to install the command line utils or leave it up to the user.

[eschwartz]

Originally Posted by KevinH

Not really, simple command line tools should be installed in standard locations and use the PATH, both Linux and Mac OS X have that right.



It is no different from what you have to do on Linux. What's your point?



Application bundles are not needed for simple command line tools. That is why a hybrid system allowing the best of both words is the correct path forward in my opinion.

Basically, that the installer, whether that be a Windows MSI, OSX Application Bundle, or linux self-extracting tarball or distro package, should be able to handle all that setup by itself, without the user needing to babysit.

And I genuinely don't know much about OSX, possibly there is some way of installing an Application bundle that also does the necessary post-install setup for complex offerings?


For calibre, it is still heavily recommended to use the standalone binaries, so I had to write my own system service for triggering updates (manually or automatically). I also ported it to OSX, although I can't really test it I have been told it works as expected.
On OSX, it symlinks the CLI tools, running as root, after mounting the DMG and copying the application bundle.

You will hate to know that the latest Mac OS X El Captian defaults to disabling root entirely (ie rootless) with no way to change that unless you reboot in a special way. root still exists, it is just that nothing can become root (sudo and su stops working) and a set of system level directories are locked in stone, unable to be touched in any way. If interested:

google El Captian System Integrity Protection



No he agrees that distribution differences and the need to recompile are both what is killing the Linux Desktop. For the Linux Desktop to succeed, the Linux distributions must agree on how to handle all of these issues or be stuck in the past so to speak.

Oh, I agree distro differences need to be resolved. It seems to me to be a linked issue with non-recompiled /opt software -- essentially, commercial, closed-source software MUST be able to install in an easy, cross-distro manner with the same stability as on Windows and OSX.

Sandboxing, containerization seem to be separate issues, of general security. Although also good things to have.
Drag-and-drop install is just a method, one-click installers should not be fundamentally more challenging to the Linux Desktop.

DEBs and RPMs I believe can already prompt you to install them, and you can put a standalone /opt -contained application in it, but of course the main issue is multiple competing standards.

Then he has changed his mind. Many many years ago I remember reading a rail of his against the damn registry nonsense. I will look for the link.

Well, I think the issue may have been more about the single point of failure, with a binary registry that is the source of all information.

There has to be some sort of consistent way of figuring out what opens what and which applications are available.

Linux doesn't have a registry, per-application desktop files are the metadata that says which applications do what, and and there are curated lists of mimetypes ("shared-mime-info").
The mime database reads all those and caches the information, and actually throws it all away and recreates it from scratch IIRC every time a package post-install scriptlet triggers the update-{mime,desktop}-database command.


It's similar to the way system logs are plain text files, and why systemd bothers so many people.

I am not disagreeing. I think Windows could make some improvement here as well. As could Mac OS X by the way. No Desktop system is perfect, but a lot of new ideas have come forward recently and it seems like no one on the Linux side sees their value to help unify and fix the Linux Desktop.



Agreed. Getting the different Linux distributions to make any consistent changes (or even decide on a standard Desktop user experience!) has been like herding cats. It simply isn't going to get done until someone like Linus pushes for it. At least, he is now seeing it as the problem it has always been.



Yes but I am hoping for much more from Linux. They have a chance to set up something really good here if they can just focus and get something done. I am a big user of Linux but not for Desktop use. I use it to house my research data, to create software to manipulate my data and run statistical analyses. Almost all via ssh. I would love to be able use Linux on the Desktop in some sane way that did not require me to basically become my own maintainer.



Not exactly. All of these are related to the user experience and as a developer, I have to worry about and deal with all of this nonsense, as do you.

Take care,

KevinH

Yep -- this about sums it up.
"I would love to be able use Linux on the Desktop in some sane way that did not require me to basically become my own maintainer."

Until that gets solved, we will probably just see a few things, available specifically against "whatever the current stable version of Ubuntu is".


For Windows, the Windows Store is trying to solve this, but I like the look of Chocolatey myself.
So does Microsoft, they modeled OneGet after it.

And of course OSX curates applications and tools through homebrew/cask.