Layer7 API Management

Hi Techies ,

We are able to see Cassandra write query in our environment in case of high load , Please have a look over below log snippet .

Error String :

WARNING 8511 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

As per error ,looks like it Its not getting ack in time from both the replica  hence timing out with above error  But in our environment there are 4 Cassandra node and all are running fine with out any issue .

Software version that we are using :

Gateway :9.2 Gateway  2 node standalone (A and B)

Cassandra : 5.1  Total 4 nodes  Cassandra  (A,B ,C and D)

OS : REHL 6.7

Thanks!

Hi

Did you set any custom value's for the cassandra connections ?

see also :

Cassandra Cluster Properties - CA API Gateway - 9.4 - CA Technologies Documentation 

Regards Dirk

Thank you Dirk for checking !!

No we did not set any custom value in Cassandra connection 

Dear Prashant SRIVASTAVA ,

This should be an error from the DB server.

Since it's due to timeout, can you try increase the write timeout of Cassandra?

( I am not 100% sure as I don't have cassandra, but it should be in the conf/cassandra.yaml file.

write_request_timeout_in_ms: 30000

)

It seems you configured replication for cassandra, then the default write timeout may not be enough.

Regards,

Mark

Thank you for Reply Mark!!

We are using default read and write  timeout at db level that is defined in Cassandra.yaml file .While digging more in policy manager audit log can see there are two type of error related to write query  . As per below log seems API Gateway dynamically sending consistency level(TWO and LOCAL_ONE) with query but i did not get in Policy any such configuration .[all Cassandra node are up

 and running fine ] 

sensitive)
2018-12-20T09:25:41.219+0100 INFO 11509 com.l7tech.external.assertions.comparison.server.ServerComparisonAssertion: 7101: Comparison did not match: ${tempRefreshToken} is not empty (case sensitive)
2018-12-20T09:25:41.219+0100 WARNING 11509 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write

2018-12-20T09:25:41.460+0100 WARNING 16161 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)
2018-12-20T09:25:41.511+0100 WARNING 10155 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency LOCAL_ONE (1 replica were required but only 0 acknowledged the write

Thanks!

Prashant Srivastava

Hi ,

I am not aware of a setting in the gateway for this , if you see different consistency levels being set from the GW .

I would suggest to raise a support case so we can have closer look and investigate the logs .

Dirk 

Hi 

One other thing which could play a role which OTK kit are you using , and how is it setup 

see:

Apache Cassandra Database - CA API Management OAuth Toolkit - 4.3 - CA Technologies Documentation 

During these steps you set the replication factor . which could explain why you see different consistency levels

Dirk 

Thank Dirk for looking in to it!!

I can see replication factor configured as 2 in our environment .But not getting how can see different consistency level ((TWO and LOCAL_ONE)) in Gateway log . Also in what type of  scenarios we can see different CL level . 

CREATE KEYSPACE otk_db WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = true .

I have already logged case [01254681 ] on our Support Portal.

Hi 

I will keep a eye on the case but it seems the replication factor 2 is coming form a otk update .

Which could be set in the during  database creation for the otk kit.

Apache Cassandra Database - CA API Management OAuth Toolkit - 4.3 - CA Technologies Documentation 

Dirk

Hope you are doing well Dirk ! 

Did you get anything more on this . Also i was looking in to below log snippet and can see code like 8511 after WARNING , any idea what does it mean ?

WARNING 8511 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

While going through documentation got to know there is few audit message code but  I did not get 8511 in the list .

FYI

Audit Detail Codes - CA API Gateway - 9.1 - CA Technologies Documentation 

Thanks!

Prashant Srivastava

adding on this if number present there in the log is related to Audit Message Code then it should be constant but I can see its different in below logs .

2019-01-08T10:27:19.888+0100 WARNING 354 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:31.788+0100 WARNING 7466 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:41.124+0100 WARNING 7393 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:42.201+0100 WARNING 11417 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:43.475+0100 WARNING 7393 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:44.673+0100 WARNING 11417 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:45.796+0100 WARNING 7393 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:47.084+0100 WARNING 11417 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:47.424+0100 WARNING 5965 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

2019-01-08T10:27:47.769+0100 WARNING 7139 com.l7tech.external.assertions.cassandra.server.ServerCassandraQueryAssertion: 10903: "Perform Cassandra Query" assertion failed due to: Cassandra timeout during write query at consistency TWO (2 replica were required but only 1 acknowledged the write)

Hi

The 8511 in your example is not the audit detail code that would be the 10903.

Did you check the support case ?

Our consistency level by default will use ONE and it is not going to be configurable until 9.5+

When the batch statement is run it will require at least 2 nodes in the Cassandra cluster available so the key part will be to determine of the cassandra nodes how are they separated into DC.

Regards Dirk