Symantec IGA

These  steps should help address this problem as noted in recent advisory quickly

Symptoms if expired out of box certificates not been addressed

• New provisioning server/connector server connections or connection updates for example when acquiring new endpoints may not work
• If you have made a change on any of your IDM servers and then restart your server for the change to take effect, you may see server restart may not happen due to failing connections to the provisioning/connector servers etc. 
• You may see the following messages in your IM application server logs:

      Caused by: java.security.cert.CertPathValidatorException: timestamp check failed 
         Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sat Nov 25 01:26:01 MST 2017

• You may see the following messages in your DSA warn logs:

      [56] 20171127.100608.055 WARN : Certificate 'D:\SMA\CA\Directory\dxserver\config\ssld\personalities/hostname-imps-router.pem'       is outside of validity date range

      [56] 20171127.100608.055 WARN : Unable to get certificate from       'D:\SMA\CA\Directory\dxserver\config\ssld\personalities/hostname-imps-router.pem'

      [56] 20171127.100608.055 WARN : set_cert_stuff failed

      [56] 20171127.100608.055 WARN : Cannot get personality

How to Confirm if certificates are out of the box and have expired

1) Provisioning certificates

On each provisioning server, running the below command will show if the expiration date is Oct 6.

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\data\tls>..\..\bin\openssl x509 -enddate -noout -in et2_cacert.pem

WARNING: can't open config file: /usr/local/ssl/openssl.cnf

notAfter=Oct  6 08:25:50 2017 GMT -> in this example, this means certificate expired on Oct 6

2) DSA certificates

------------------------------

On the machines where the Provisioning Directory is installed, open a command prompt and run the command:

dxcertgen report

This command will list all the certificates and their validity dates. Expired certificates will be marked as invalid. Here is an example:

- <hostname>-impd-notify.pem -
certificate : 1
version     : 3
serialNum   : 311
issuer      : /C=US/ST=NY/L=Islandia/O=Identity Management/OU=Provisioning Servi
ces
notBefore   : Nov 28 18:26:00 2007 GMT
notAfter    : Nov 25 18:26:00 2017 GMT
subject     : /C=US/ST=NY/O=Identity Management/OU=Provisioning Services/CN=eta_
server
status      : *** INVALID ***

- <hostname>-imps-router.pem -
certificate : 1
version     : 3
serialNum   : 311
issuer      : /C=US/ST=NY/L=Islandia/O=Identity Management/OU=Provisioning Servi
ces
notBefore   : Nov 28 18:26:00 2007 GMT
notAfter    : Nov 25 18:26:00 2017 GMT
subject     : /C=US/ST=NY/O=Identity Management/OU=Provisioning Services/CN=eta_
server
status      : *** INVALID ***

If the certificates have not expired, take note of the notAfter date for future reference.

Steps to replace certificates

If certificates are OOTB and have expired, proceed as follows:

• Note the attached ootb_certs.zip (relevant only for IDM 12.6X releases) and ootb_certs_SHA1.zip (relevant only for IDM 12.5X releases) on this post. 

 Replace Provisioning Server Router DSA certs

On each Provisioning Server (where you typically have imps-router DSA running):

• Go to" Attach" towards bottom of post, and extract the relevant zip attachment (either ootb_certs.zip if on 12.6X or ootb_certs_SHA1.zip if on 12.5X) on this post.
• Navigate to the pd folder and copy the impd_trusted.pem file to DXHOME\config\ssld location, and overwrite the existing one.
• From the same pd folder, rename the provided imps-router.pem to reflect the actual  local hostname, and copy that into your DXHOME\config\ssld\personalities location and overwrite the existing one.
• Delete any other .pem files related to 'imps' and 'impd' you have in there.
• Basically on each Provisioning Sever host, you will end up with only one router .pem file reflecting the local router name.
• Restart your DSA performing 'dxserver stop all' followed by 'dxserver start all' command.

Replace Provisioning Directory DSA certs

On each Provisioning Directory Server (where you typically have impd-main, impd-inc, impd-co and impd-notify DSAs running):

• Take the same impd_trusted.pem used above in 1), and copy it to your DXHOME\config\ssld location and overwrite the existing one.
• From that same ootb_certs.zip/ootb_certs_SHA1.zip extraction and pd folder, rename the provided impd files (ex. hostname-impd-co.pem) to reflect your local data DSA names, and then copy the files into your DXHOME\config\ssld\personalities location and overwrite the existing ones.
• Delete any other .pem files related to 'imps' and 'impd' you have in there.
• Basically on each Provisioning Directory host, you will end up with only four impd .pem files reflecting the four local data DSA names.

• Restart your DSAs performing 'dxserver stop all' followed by 'dxserver start all' command.

Replace Provisioning Manager Certs

For Prov Manager you replace in two places.

1) from package path "prov/data/tls/" -> on the host under <Provisioning Manager>/data/tls/

2) from package path "prov/data/tls/client/ -> on the host under <Provisioning Manager>/data/tls/client

3) Restart Provisioning Manager.

Replace Provisioning Server Certs

For Prov Server you replace in just one place.

1) from package path "prov/data/tls/" -> on the host under <Provisioning Server>/data/tls/

2) Restart Provisioning Server.

Now you can follow information in Update Your Provisioning Certificates - CA Identity Manager - 12.6.8 - CA Technologies    Documentation starting at:

• Java Connector Server
• Connector Xpress

NOTE: For both of the above, if you are running Java/JRE 1.5, the provided keytool command in the documentation will not work as that version doesn't support '-importkeystore' option. Your workaround would be to upgrade Java/JRE to at least 1.7 and the command should work.

• Connector Server SDK
• Update jiam.jar File (Ensure you follow the right Use Case 1 or 2 depending on your IDM release)

NOTE: 'Use Case 2' also applies to IDM 12.5X release (or you can use this TEC1561732 for the same)

Thank You Palaka for the steps. There are lot of confusions among the customers about this update, Could you also clarify the below ?

-  Does this applicable to the customers who does use SSL communication in the provisioning directory/server ONLY ? or regardless of communication protocol everyone has to apply this update ?

- Also what all are the IDM component's communications are getting affected by this cert expiry ? for example, IDM talks to Provisioning directory which could be SSL or NON-SSL which is configured in IDM Manage console. what else ?

Appreciate your clarification on this.

Thanks

Ashok

Ashok,

The main link where download and instructions are provided actually lists out what components of Identity Manager this is applicable to.

A quick example would be, Provisioning Manager will fail to start as it will use SSL to authenticate to local Provisioning Router DSA to chain the request to remote Provisioning Directory (impd-main) DSA due to expired root CA. If Provisioning Server is offline you can image what other cascading problems it will bring.

For now (due to lack of time), I would recommend everyone should look at the main link, see what components are effected and use their judgment to make a decision on whether this is applicable to their specific setup or not.

I know this is not a detailed answer but hope this helps somehow.

Thanks,

Hitesh

Additionally, this problem could cause the CA Directory replication for the Provisioning Directories to get out of synch.

In case CA Directory Multi-write (MW) recovery is needed here is a doc and instructions:

It appears that you may need to perform a MW Data recovery.

See Page 9:

http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec479507.aspx

Select a DSA that is completely up to date, as the one that you are going to copy the data from.  In the below example I'm using the 'dx1' DSA as the DSA that IS up to date.

Please follow the below recommended steps which was created as DSA 'dx2' (Host 2) being the corrupt DSA, and 'dx1' (Host 1) being the up to date DSA:

The following commands would need to be performed via command prompt.

For Linux/Unix environments, you will need to su to the dsa user (su - dsa) before running the commands.

- Shutdown DSA 'dx2' (dxserver stop dx2)

- Backup / Delete existing db, dp and tx file ($DXHOME/data) for the 'dx2' DSA.

- Perform dxdisp on Host1 for Host1 (by running the follow command: dxdisp dx1)

- Perform dxdisp on Host1 for Host2 (by running the follow command: dxdisp dx2)

- Perform dxdisp on Host2 for Host2 (by running the follow command: dxdisp dx2)

- Perform dxdisp on Host2 for Host1 (by running the follow command: dxdisp dx1)

- Copy over dx1.DB file ($DXHOME/data) from Host 1 to the same location on the Host 2 server.

- Rename DB file on Host 2 accordingly (renamed the copied database from dx1.db to dx2.db).

- Start DSA 'dx2' (dxserver start dx2)

- Confirm MW is working properly.

Hi Guys,

I have taken over support for our IDM Environment and looking at the documentation provided by CA I am going to have to run the keytool to update our Java Connector Server:

1. At the command prompt, navigate to the above folder location and run the following command:

keytool -v -importkeystore -srckeystore eta2_server.p12 -srcstoretype PKCS12 -destkeystore ssl.keystore -deststoretype JKS

2. Specify the following passwords when prompted: destkeystore password Value: Java Connector Server password. srckeystore password Value: secret Note: Type "yes" when prompted to overwrite the server certificate.

I can not find the initial destkeystore password is there a way I can find this in the config files located: H:\Apps\CA - Backup\Identity Manager\Connector Server\jcs\conf?

Musangwe,

   in our docs we call out a default password:

Update Your Provisioning Certificates - CA Identity Manager - 12.6.8 - CA Technologies Documentation 

   Have you tried with the default password? step 5 under connector express.

Bill

Hi Bill,

Thanks for the response I was able to locate the passwords.

But I have another question in the documentation under Connector Xpress it talks about updating the keystore:

1.    Ensure that Connector Xpress is closed.

2.    From the downloaded certificate zip file, copy the eta2_server.p12 and eta2_client.p12 files from the conxp folder.

3.  /conf

We only have two certificates both expired:

·        Ca_fips

·        Root_ca

But when I check the location

Kind regards,

Musangwe Kalowa

AMS Identity Access Management Consultant

If CA SSO (Siteminder) is involved and customer has implemented full integration we must also update the certificate in the CA SSO (Siteminder) key database:

CA SSO (Siteminder)

Copy the new impd_trusted.pem from the provisioning server to the Policy Server.

Import the new pem file from the certutil in SiteMinder/bin (your paths may differ)

the name (-n below) of the cert may differ in your environment as well

Lists the certificates in the store
certutil -L -d C:\certdatabase

This shows information specific to the certificate named "Provisioning Certificate" such as valid until date:
certutil -L -d C:\certdatabase . -n "Provisioning Certificate"

Updates the certificate with the cert copied from the prov server:
certutil -A -n "Provisioning Certificate" -t "P,," -i C:\certificates\impd_trusted.pem -d C:\certdatabase

Then verify the new cert:
certutil -L -d C:\certdatabase . -n "Provisioning Certificate"

Hi Bill,

Thanks for this information I believe our environment is fully integrated IAM 12.6.02 and Siteminder  12.52 sp01 cr04.

I have read through some documentation about verifying if enviromenmtns are integrated and I came across one that ask one to look for the following:

"d="cf128b97-1ec6-11b2-82a1-ca5535a7a489" version="126.2.0.213" copyright="2011" info_url="" support_url="http://supportconnect.ca.com" location="C:\Program Files (x86)\CA\Identity Manager" last_modified="2013-10-18 10:23:12">
   <vendor name="CA" id="3b7cec0e-1ec0-11b2-968e-ca5535a7a489" home_page="http://www.ca.com" email=""/>
   <feature name="Extensions for SiteMinder (if SiteMinder is installed locally)" last_modified="2013-10-18 10:24:29">
   <![CDATA[IdentityMinder Extensions files for Policy Server]]>
    <component ref_id="e8f35342-1ec6-11b2-84a6-ca5535a7a489" version="126.2.0.213" location="C:\Program Files (x86)\CA\Identity Manager\.\install_config_info\im-uninstall\uninstall.exe"/>
    <component ref_id="802fbd5d-1ee7-11b2-ba58-8ff1afdfe42a" version="126.2.0.213" location="C:\\AppData\Local\Temp\3\591755.tmp\framework.properties"/>
    <component ref_id="49055ea0-1ee8-11b2-9cff-8ff1afdfe42a" version="126.2.0.213" location=""/>
   </feature>  "

https://support.ca.com/us/knowledge-base-articles.TEC535804.html

But When following your work instructions I have tried listing all the cert on our Siteminder Policy servers and I am unable to locate and certdatabase.

I also went further and looked for the following three files:

1. a Cert database with the following three files:

cert8.db

key3.db

secmod.db

I cant locate them either.

does that mean our environment is not fully integrated or just doesn't have the Certdabase, if so do I need to install one?

Musangwe,

    Again, I have to point to documentation for this:

https://docops.ca.com/ca-single-sign-on/12-7/en/configuring/policy-server-configuration/configure-policy-server-data-storage-options/configure-an-ssl-connection-to-an-ldap-data-store

    I do not know your SM version, so this is for 12.7, no changes since 12.52.

under the heading

Create the Certificate Database Files

step two says:

Enter the following command:

certutil -N -d certificate_database_directory

• -N
  Creates the cert8.db, key3.db, and secmod.db certificate database files.
• -d certificate_database_directory
  Specifies the directory in which the certutil tool is to create the certificate database files.

if you have access to the policy server management console you can check the data tab

If there are still troubles after this post, please open a support ticket to help.

Hi Bill,

Thanks for all the help.

No we do not have a certdatabase file I checked Policy Management console and that value is blank.

Our siteminder environment is 12.52 it is using ODBC as storage and AD as the user store.

Kind regards,

Musangwe Kalowa

AMS Identity Access Management Consultant

Hi Palaka,

 i have performed all the steps you mentioned.

But when i am trying to start the provisioning server .

The below message is appearing 

"im_ps failed to start within 120 seconds"

Please help me on these

Hi SabreeW75350521,

This cannot be worked out in community thread as there could be many things that we might to check and go back/forth. I suggest you open a support case.

STATUS UPDATE SINCE LAST COMMUNICATION:

Due to this problem, an installation or upgrade to existing 12.x SPs or CRs will fail.  As a result, we have replaced the following on our Downloads and Docops sites:

12.6 SP4 CR4

12.6 SP5 CR2

12.6 SP6 CR1

12.6 SP7 (12.6.7)

12.6 SP8 (12.6.8)

12.6 SP8 CR1

The only changes made to these versions are updates to the certificates related to the CA Directory in the provisioning server and provisioning directory.  There are certificates for the other components, including the provisioning server, that will still need to be manually updated after install or upgrade.  (Follow instructions below in the Problem Resolution section to replace the certs after upgrade/install.)  

If you have downloaded any of these prior versions, please discard them and use the new downloads for an upgrade or fresh install.  If you have already replaced your certs, no action is needed unless you plan a new install or upgrade to one of these versions.  The Docops pages for each of the refreshed CRs have been updated with instructions on how to determine the difference between the original and updated CRs.

SPs can be located on http://support.ca.com Download Management area.  CRs are on http://docops.ca.com

We have Provisioning Certificate Utilities to enable you to verify whether the CA Identity Manager components are using the new certificates.  There is also a link to these for each of the CRs on Docops and can be found here: https://docops.ca.com/ca-identity-manager/12-6-8/EN/upgrading/upgrade-provisioning-components/ca-identity-manager-certificate-utilities

(The instructions and downloads are the same for each SP and CR)

Communities post on this topic to follow: https://communities.ca.com/thread/241785381-steps-to-address-expired-6-oct-2017-provisioning-certificates-in-identityminder

 
PROBLEM RESOLUTION:

Follow the instructions in the Important Notice in the documentation set for your release at:

CA Identity Manager 12.6.08 (SP8) - https://docops.ca.com/ca-identity-manager/12-6-8/EN/release-information/release-notes-12-6-08-cumulative-patches

CA Identity Manager 12.6.07 (SP7) - https://docops.ca.com/ca-identity-manager/12-6-07/en/release-information/release-notes-12-6-07-cumulative-patches

CA Identity Manager 12.6.06 (SP6) - https://docops.ca.com/ca-identity-manager/12-6-6/en/release-information/release-notes-12-6-06-cumulative-patches

CA Identity Manager 12.6.05 (SP5) - https://docops.ca.com/ca-identity-manager/12-6-5/EN/release-information/release-notes-12-6-05-cumulative-patches

CA Identity Manager 12.6.04 (SP4) –  https://docops.ca.com/display/CIM12604/Release+Notes+-+12.6.04+Cumulative+Patches

If you are using a release prior to 12.6.04 (SP4), please refer to the instructions for 12.6.04 (SP4) as these instructions are consistent and apply to all prior releases.

For End of Service version 12.5, please see the following tech doc:

https://support.ca.com/us/knowledge-base-articles.TEC1561732.html

If you have any questions about this Critical Alert, please contact CA Support.
 
Thank you,

CA Support Team
1-800-225-5224

http://support.ca.com

Latest Notification:

This is a follow up to prior notifications sent out about the CA Identity Manager certificate expiration that occurred on Oct. 6^th.    If you have followed all of the steps provided in the previous notification and restarted the Identity Management, provisioning and directory services there should be no issue. Please note that as part of the instructions previously provided there are Directory certificates that also need to be replaced.  Since these expired on Nov 25^th, please double check these in case they were missed.  If you restart and have not replaced all of the certs problems will arise.  Please see below for further information.

PRODUCT(S) AFFECTED: CA Identity Manager                      RELEASE: 12.x

PROBLEM DESCRIPTION:

The CA provisioning server ships with out of the box certificates, including Directory DSA, that for version 12.x were due to expire on 6th October 2017 and Nov 25^th 2017. If you are using your own certificate or have deployed 14.0 or later, this does not affect you, as from 14.0 GA, the provisioning certificate shipped with the product has been updated with a newer one.

POTENTIAL SYMPTOMS:

New provisioning server/connector server connections or connection updates for example, when acquiring new endpoints may not work

If you have made a change on any of your IDM servers and then restart your server for the change to take effect, you may see server restart may not happen due to failing connections to the provisioning/connector servers, etc. 

            You may see the following messages in your IM application server logs:

Caused by: java.security.cert.CertPathValidatorException: timestamp check failed. Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sat Nov 25 01:26:01 MST 2017

               You may see the following messages in your DSA warn logs:

[56] 20171127.100608.055 WARN : Certificate 'D:\SMA\CA\Directory\dxserver\config\ssld\personalities/hostname-imps-router.pem' is outside of validity date range

[56] 20171127.100608.055 WARN : Unable to get certificate from  'D:\SMA\CA\Directory\dxserver\config\ssld\personalities/hostname-imps-router.pem'

            [56] 20171127.100608.055 WARN : set_cert_stuff failed

[56] 20171127.100608.055 WARN : Cannot get personality

IMPACT:
For any 12.6.08 (SP8) and earlier releases, if you are using expired certificates, it will cause any of the following: requests being sent to a provisioning server to fail; directory replication failure; provisioning server service not starting; connector server requests failure; IM environment fails to start; IM provisioning directory communication failure.

PROBLEM RESOLUTION:

Please follow How to Confirm if Certificates are out of the box and have expired section at the following post: https://communities.ca.com/thread/241785381-steps-to-address-expired-6-oct-2017-provisioning-certificates-in-identityminder

If expired, please follow Steps to Replace Certificates section in the same post.

Alternatively:
Follow the instructions in the Important Notice in the documentation set for your release at:

CA Identity Manager 12.6.08 (SP8) - https://docops.ca.com/ca-identity-manager/12-6-8/EN/release-information/release-notes-12-6-08-cumulative-patches

CA Identity Manager 12.6.07 (SP7) - https://docops.ca.com/ca-identity-manager/12-6-07/en/release-information/release-notes-12-6-07-cumulative-patches

CA Identity Manager 12.6.06 (SP6) - https://docops.ca.com/ca-identity-manager/12-6-6/en/release-information/release-notes-12-6-06-cumulative-patches

CA Identity Manager 12.6.05 (SP5) - https://docops.ca.com/ca-identity-manager/12-6-5/EN/release-information/release-notes-12-6-05-cumulative-patches

CA Identity Manager 12.6.04 (SP4) –

https://docops.ca.com/display/CIM12604/Release+Notes+-+12.6.04+Cumulative+Patches

If you are using a release prior to 12.6.04 (SP4), please refer to the instructions for 12.6.04 (SP4) as these instructions are consistent and apply to all prior releases.

For End of Service version 12.5, please see the following tech doc:

https://support.ca.com/us/knowledge-base-articles.TEC1561732.html

If you have any questions about this Critical Alert, please contact CA Support.
 
Thank you,

CA Support Team
1-800-225-5224

http://support.ca.com