It is expected to log that message about "80 characters" because it is from the SAML specification.
But it does not mean we will truncate the value to 80 characters, so you would find longer relaystate still works.
I do not know what would be the maximum length siteminder can support. You can submit a case if you need the actual limit.
But as the URL can grow pretty long while federating, you might hit the 2000 chars limit on the IE browser before hitting siteminder limit(if there is).