When I get errors opening TCP VSIs, it's always because the message isn't safe to be opened as XML representation in DevTest. It means that it was a message with control characters in it, such as ^@ (0x00 or �), ^A (0x01), etc, that would cause issues if it was displayed.
Valid characters in XML - Wikipedia
You'll need to save the raw traffic during your recording process, then manually decode that traffic in a text / binary editor (like Notepad++ on Windows or TextWrangler on Mac), to determine what's happening in your messages. TCP is a lower-level protocol than something like HTTP, so it doesn't have the same strict standards as to how data is encoded - you might need to do some encoding yourself in a custom or a scriptable DPH, or investigate whether the Opaque Data Processor is relevant for your situation.