Layer7 API Management

Having just spent the best part of the morning debugging this behaviour, I thought it was worthwhile sharing with the community.

Let me first describe the situation I was finding:

- I had previously created lots of APIs with mutual SSL on the GW (require SSL with client certificate authentication).

Typically the client certs are self signed certs generated by the consumers of the APIs - the public cert being stored in a FIP and compared in the policy to allow access to a particular client.

I've done tons of these for various customers.

To test with SOAPUI, add the p12 key to the SSL options, etc and all is well - GW prompts for client cert on testing a mutual SSL API - soapui sends cert, job done.

If any of above is new to you - plenty of info on internet about setting soapui up to send client certs, etc.

Now, with a new customer, I had another mutual SSL requirement for an API - so did it again. BUT suddenly soapui is not working anymore - it is not sending the cert.

OK, that's enough background - I won't go into the 4 hours of debug - just present solution now:

- SOAPUI 5.3.0 (all all previous version afaik) has the following weird behaviour:

- If the SSG is set to allow TLS1.0 only (the default until I believe v9.0 of the GW), then SOAPUI works fine - it does the handshake, gets asked for client cert, provides it, GW verifies in policy and all is good.

-If the SSG is set to allow any other versions (1.1 only, 1.2 only, 1.1+1.2, 1.0+1.1, 1.0+1.2, 1.0+1.1+1.2) soapui will fail to work. It will not send the client cert at all and your policy will therefore fail since no client cert was received.

Now this is not strickly speaking just a soapui issue, but an interpretation it is making for RFC 2246:

https://www.ietf.org/rfc/rfc2246.txt 

When the server (the GW) asks for a cert it tells the client what cyphers it likes, and what CAs it PREFERS:

page 41:

"

    certificate_authorities            A list of the distinguished names of acceptable certificate            authorities. These distinguished names may specify a desired            distinguished name for a root CA or for a subordinate CA;            thus, this message can be used both to describe known roots            and a desired authorization space.
"

Now, by default if there are no CA capable keys in the GW private key store, this will be (wrongly imho) the server's SSL key (which in almost all cases is not a CA capable key).

IF there are CA capable keys in the GW private key store, it is set to a list of those.

What soapui seems to do ONLY honor this when it doesn't negotiate TLS1.0. If it's TLS1.0 - all is well.

NOTE: it does this even when TLS1.0 is still supported since it does 'support' TLS1.1 - so it effectively breaks itself by choosing 1.1 if it's available on the server.

From a client point of view I don't know how many clients/consumers this behaviour effects - it depends on their implementation of RFC2246 frankly. 

Worst case:

- you have mutual SSL protected APIs developed with self signed certs now - working fine with your SSL settings configured at the default of TLS1.0 only (I know this is terrible - but it WAS the default until v9.0). If a pen test, etc points out this as bad (as it should) and you disable TLS1.0 and enable 1.1 and/or 1.2 - there is a possibility that some/all of your clients will not be able to connect to your mutual SSL APIs anymore IF their client code/libraries implement RFC2246 in the same way as soapui (i.e. it's asking for cert signed by CA 'bob.com' - the one I have is a self signed cert - so I won't send it)

Best case:

- your clients all hard send the certs anyway, and effectively ignore the CA list in the servers client cert request. This is the case with command line tools for example - like wget and curl which still work fine.

NOTE: This is not a GW bug. I am simply pointing out that the vagaries of client authentication in the RFC will mean that a variety of implementation are out there consuming any mutual SSL APIs you have - so IF/when you turn off TLS1.0 expect to have to do some regression testing to determine if you are affected by the above.

OK - so you've tested - and it IS worst case - your callers code isn't sending the certs anymore. What can you do ?

Well, you need to do the following:

1. create a CA capable cert of the GW (see docs)

2. ask your customers to generate a CSR for their current client cert and send you it

3. sign the CSR in the GW PM with the new CA capable cert you have created.

4. give customers the signed cert and tell them to replace the cert chain of their key with that.

5. customer now can use same client key (but now it's signed with CA)

NOTE: no policy changes should be required since it's the same client key still - so still valid in any FIP you have it in, will still authenticate with whatever code you had in there to do that previously, etc.

Now, when the SSL handshake starts, the GW will ask the client for a cert, and will tell it that it would prefer one signed by the CA capable cert CN you just created.

Since the client cert IS now signed by a CA in that list, it will send it and all will be well.

Thanks for sharing, StuartSmith75811464!

It looks like you may not understand the root of the issue, though you did find the RFC spec...

In the TLS 1.1 and 1.2 negociation the server sends it's trusted certificate list and the client SHALL NOT use it's certificate in handshake negociation if the client's certifiate issuer (itself in the case of self-signed certifciates) is not in the server's trusted list.  When configuring services to authenticate self-signed client certificates using TLS1.1 or 1.2 you may add the client certificate (becuase it is effectivly it's own authority) into the manage certifiates list marked for use signing client certificates and as a trust anchor (depending on your configuration of the FIP in which the user with the client certifcate resides you may or may not have to add it there as well); and thats it, they'll start working.

Hi, the solution you put forward was how a client got it working in fact.

It is however, imho a poor implementation decision if you do this - hence I didn't suggest it.

You will end up typically with two copies of each and every client public cert - one in the cert store, and another in your FIP.

You now have 2 certs to keep synchronised. adding to support issues, and the chance of mistakenly getting these out of sync.

Also in the real world, these certs expire and you will be using a standard 2 FIP alternating model to update the new cert while maintaining the old one till it expires to eliminate production outages when client change their certs (since typically the CNs will be the same as must be unique within a FIP).

You will therefore have to have both certs in your cert store here as well... and something else to have to clean up.

With a large number of clients, this is going to be a nightmare to manage and imho is exactly why you should be signing these with a CA - and only having the CA in the cert store - your client certs are for identifying the client - they should be in your FIP store only.

I can see in the RFC that if the list is empty any cert can be sent, so that is perhaps another way around this issue, however I don't know how you can sent an empty list - as the GW iseems to default to sending every cert in your cert store which is pretty naff imho - exposing a whole lot of internal implementation to the caller (e.g. a list of all your customers).

At the very least the list of allowed client certs should be configurable per service on the GW, but this does not appear to be possible.

Take an example of an APIs secured by Mutual SSL. Each one will have a client cert with a CN probably identifying one of your customers by name (you might have 100s of these).

Your way exposes every single one of those customers names when anyone calls this API -

SSL handshake: 'GW: hi - please send me a cert with a CN matching one of this massive list of every one of my customers (....).'

not great imho - your competitor now knows who to talk to. It exposing implementation and security details of your service. 

My method again, will not expose all of this potentially valuable commercial information.

SSL handshake: 'hi - please send me a cert SIGNED by our CA (xyx)'

they get no info about your customers.

It does however add some overhead, in that clients must present there cert CSR to you for signing, but that still seems far less complexity and duplication than your method. 

However the spec DOES allow for an empty list which would allow existing solutions to work with no changes whatsoever - with certs simply being in the FIPs as worked for TLS1.0 - perhaps there is a way of controlling the certificate request/certificate authority list in the SSL handshake on the GW and someone from CA can explain it ? If there isn't I'd suggest that it would be a useful additional configuration option - all within the RFC.

hope that's useful to some people.

I did not mean to imply that your solution wouldn't work, just that adding them to the trusted list was less complicated.

Yes, we do end up maintaining two copies of thier certifcate.  To be fair we have the expiration issue for most client certificates because the majority of them will have a trusted third party issuer (unless we're talking about internal consumers in which case we have an internal PKI managed by another group, which is effectivly what your becoming for yourself).  Since the gateway sends it's trusted cert list, to make it empty I think you would have to empty that list, but then the gateway wouldn't trust any backend servers for https...

The most direct way to restrict access to a list of clients is to use the authenticate against user assertion (repeat within an at least one must be true as required), which means that each user's certifciate has to be in a FIP anyway (or you could use a compare on the name after your authenticate against authority, but then you have to maintain your user restriction effectivly as a text filter list of compare assertions).  Yes, we have hundereds of certs to maintain, it's part of the job.  Yes, in the handshake the self-signed client's are present for everyone that calls the gateway on a port with mutual ssl permitted; which is just one reason that we do not allow self-signed certificates in production (and if a partner can generate a csr then they can go get it signed by a trusted third party instead of us).

Your method does have one more issue though, your customers now have a certifiate signed by you that you do not have contol over whom they send it to, which could give a third party the impression that your customer is acting with your authority (because they have a certificate signed by your organization) which could open you up to liability or bad press through no further action of your own.

So the bottom line is; both of these work, but neither is advisable.

Hi Stuart and Ben, I also had to figure out the cause of this error the hard way. Luckily though, there is a simple way to configure the Gateway to send an empty distinguished-names list in the server Certificate Request. This applies to TLS 1.0, 1.1 and 1.2. Gateway version 9.2 is the recommended base-version for this use-case.

For Gateway v9.2: On the listen port's 'Advanced' tab, add an advanced property: noAcceptedIssuers=true. This should take effect right away.

This functionality is also possible on Gateway versions prior to 9.2. But since it comes with significant caveats, 9.2+ is recommended for this use-case.  

On Gateway v9.1 and before: this requires the change already noted above and an additional change of switching the TLS-provider from SSL-J to SunJSSE. This is also done with an advanced property on the listen port: protocolProvider=SunJSSE. The caveat is that changing the TLS-provider is not a standard configuration and the Gateway admin will have to weigh this decision. This is usually only considered as a last resort to solve a significant problem but I'm including it here for completeness on this topic. If any issues arise from using this config, it's likely that support will recommend upgrading to 9.2.

Thank you both for detailing your experiences around this. It gives a good picture to think on where the Gateway can improve it's mutual-auth story.

Regards,

Jordan

Engineer, CA API Gateway

Jordan,

   Very cool, per IETF RFC 5246 section 7.4.6:

If the certificate_authorities list in the certificate request
message was non-empty, one of the certificates in the certificate
chain SHOULD be issued by one of the listed CAs.

Therefore clients should send their certifciates reguardless of issuer (similar to TLS 1.0 behaviour).  I will definatly try this out.

Thanks for linking the TLS 1.2 RFC.  I also double-checked that and the other TLS version RFC's before my original reply.  Anyways yes, please report back if you get to try it out.

Considering that this is a valid configuration per the RFC and clearly a desired use-case (by this thread), I'll be thinking of ways to make this troubleshooting and configuration easier.