I believe the last release of convertlit actually contained decryption code. However, you need the private part of the public/private key for the user, which is extracted by a closed-source app.
The encryption is standard RSA. If Microsoft had been more open (yeah... riiight), any publisher could have produced encrypted LIT titles without compromising security.
Instead, LIT files are sent using an ActiveX component which provides the public key encrypted with some other system that only Microsoft software (Digital Asset Server) can decrypt -- but you can use the activex control to harvest the activated email address, which is "protected" by a ROT13 equivalent substitution, assuming you - get people to visit your web site, with IE, and Reader installed and allowing safe-for-scripting ActiveX controls.

At one point, I was thinking that it'd be neat to offer an alternative to microsoft's server-side component for the ebook sites that didn't want to give overdrive a large cut.