Since you are dealing with binary formats (outside the range of ASCII characters), you will most likely need to create a custom TCP delimiter. Sorry, I do not have an example TCP delimiter class to share but this link might help in getting an understanding of the format: How to create a custom request/response delimiter in CA DevTest
Out of curiosity, does the incoming request payload contain a 4 byte record descriptor word (RDW) that gives the length of the request payload? If so, your custom delimiter can find the length of the payload by parsing the RDW.
e.g, using a LIST of the bytes (ba),
byte[] length = new byte[4];
length[0] = 0x00; // if mainframe ba[0] probablycontains a 0x80 so make it 0x00
length[1] = ba[1];
length[2] = ba[2];
length[3] = ba[3];
int theRcdLength = java.nio.ByteBuffer.wrap( length ).getInt();
Or, if looping over the byte list, you could do some conversion to compare for the 0xFF and 0xEF.
e.g.,
if ( bytes.get(i).intValue() == 0xff ) {
mEndOfRequest = i + 2;
mStartOfNextRequest = 1;
return true;
}