I tried your patch (I reference it above), compiled & installed but it did not work for me. X still looked for a vt.
Perhaps the problem is debian vs. ubuntu? though I can't believe the package would be significantly different. The patch was applied to the source code supplied by xorg-server which if I recall does not supply Xfbdev.
Not sure what else to try.