To my knowledge Amazon hasn't shared their reasons for developing KFX. My speculation is that the primary reason was a desire to have a common rendering engine that they had full control over and that could be used in all Kindle apps and devices.
KF8 was never released for the iOS app. Instead that supports AZK, which uses the same browser-based rendering engine as
Kindle Cloud Reader. My best guess is that the modified Webkit used for rendering KF8 was not allowed by Apple. Apple has a requirement that HTML be rendered only using the Safari engine on that platform. By switching to a non-HTML-based format they have bypassed that restriction.
The current system of three formats with increasing capabilities (MOBI7, KF8 and KFX) seems pretty stable to me. I don't see any particular advantage for Amazon to do away with the older formats as long as there are a significant number of customers buying books and reading them on older devices.
On the other hand I also expect that there will be an increasing number of books going forward that are only available in KFX format. Currently Indic language books are only available that way. I have seen signs that Arabic language books and books with MathML will be coming soon and I believe that these will also be KFX only. Still I expect KFX-only books to be a minority for the foreseeable future.