Here are instructions for SM r12.5. It should be the same on r12.52. JBoss uses the Java key store, so we use the java "keytool" utility to generate keys and cert requests and import certs.
I'm assuming the company name is "MyCompany". Replace it, and other identifying information as appropriate.
Go to D:\Program Files (x86)\CA\SiteMinder\adminui\server\default\conf and back up keyStore.jks
Start a command prompt as administrator and go to D:\Program Files\Java\jdk1.6.0_45\bin
Run the following command to delete the current keypair and certificate.
keytool -delete -alias tomcat -keystore D:\PROGRA~2\CA\SiteMinder\adminui\server\default\conf\keyStore.jks -storepass changeit -v
Then run the following command to create a new key pair and self-signed certificate (ensure that the CN matches the FQDN of the server). The self-signed certificate will be overwritten in a later step.
keytool -genkeypair -alias JBoss_Key -keyalg RSA -keysize 1024 -sigalg SHA1withRSA -dname "CN=server01.mycompany.com,OU=IT Security,O=MyCompany,L=Dublin,S=California,C=US" -keypass changeit -validity 7300 -keystore D:\PROGRA~2\CA\SiteMinder\adminui\server\default\conf\keyStore.jks -storepass changeit -v
Then run the following command to create a PKCS#10 certificate request.
keytool -certreq -alias JBoss_Key -sigalg SHA1withRSA -file D:\PROGRA~2\CA\SiteMinder\adminui\server\default\conf\server01_certreq.p10 -keystore D:\PROGRA~2\CA\SiteMinder\adminui\server\default\conf\keyStore.jks -storepass changeit -v
Ask the PKI Administrator to certify the request. When you’ve received the certificate, run the following command to import it (this overwrites the previously created self-signed certificate with the certificate provided by the PKI administrator)
keytool -importcert -alias JBoss_Key -file D:\PROGRA~2\CA\SiteMinder\adminui\server\default\conf\server01.cer -keystore D:\PROGRA~2\CA\SiteMinder\adminui\server\default\conf\keyStore.jks -storepass changeit -v
Go to D:\Program Files (x86)\CA\SiteMinder\adminui\server\default\deploy\jbossweb.sar and edit server.xml
Change
keyAlias="tomcat"
to
keyAlias="jboss_key" (all lower case)
Restart the SiteMinder Administrative UI service