Pin 4 of J23 is such a "Known good ground"
You can see that the pad is part of the component-side ground plane in the earlier picture.
Finding the technical data sheet on the adapter is a bit more complicated:
(Page down a bit, well, a whole lot)
That adapter board is intended for use from 1.8v to full 12v RS-232 levels with 16v tolerance.
The serial adapter's ttl communications are set by programming the on-board eeprom using the special Win-Driver that is available with the purchase of the adapter.
See the OneFrog distributor pages and the attached file.
I suspect you skipped that part of the procedure (programming the levels to 1.8v with the special Windows driver.)
That is probably how someone else could use the board and you couldn't - they did not skip the "program it to correct voltage levels" step.
The most likely situation now: Your serial port on the SoC is fried.
The Soc is a $10 part (minimum order of 10 units) - either figure out how your going to replace it without SMD re-work tools or ship the Kindle back home as "broke".