Layer7 API Management

We are experiencing the DatabaseReplicationMonitor problem described in Knowledge Base Articles.  We followed the steps described in the article, but the problem persists.  Shown below are the user grants from our two servers, after having followed the instructions and restarting the servers.  What steps should we take next?  How badly does this issue affect performance?

xx-apiprod1

+----------+-------------------------+-------------------------------------------+
| user | host | password |
+----------+-------------------------+-------------------------------------------+
| root | localhost | *78377EDD3733CC2048B9FBB4B0E0299C8720D4FA |
| gateway | xx-apiprod2.thebank.com | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | % | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | localhost | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | localhost6 | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | localhost.localdomain | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | localhost6.localdomain6 | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| repluser | xx-apiprod2.thebank.com | *D98280F03D0F78162EBDBB9C883FC01395DEA2BF |
| repluser | xx-apiprod1.thebank.com | *D98280F03D0F78162EBDBB9C883FC01395DEA2BF |
| gateway | xx-apiprod1.thebank.com | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
+----------+-------------------------+-------------------------------------------+

MN_APIPROD2

+----------+-------------------------+-------------------------------------------+
| user | host | password |
+----------+-------------------------+-------------------------------------------+
| root | localhost | *78377EDD3733CC2048B9FBB4B0E0299C8720D4FA |
| gateway | xx-apiprod1.thebank.com | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | % | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | localhost | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | localhost6 | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | localhost.localdomain | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| gateway | localhost6.localdomain6 | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
| repluser | xx-apiprod1.thebank.com | *D98280F03D0F78162EBDBB9C883FC01395DEA2BF |
| repluser | xx-apiprod2.thebank.com | *D98280F03D0F78162EBDBB9C883FC01395DEA2BF |
| gateway | xx-apiprod2.thebank.com | *B3A15E4C40504F2D5B03FC184F764CB791BBCA02 |
+----------+-------------------------+-------------------------------------------+

Hi jcrivkin,

The default user for replication is repluser, can you run sql below to check the permission of repluser?

show grants for 'repluser'@'xx-apiprod1.thebank.com';

show grants for 'repluser'@'xx-apiprod2.thebank.com';

Also, there is a script to setup the user for replication,

/opt/SecureSpan/Appliance/bin/add_slave_user.sh
For more details, refer to Configure Replication - CA API Gateway - 9.1 - CA Technologies Documentation 

Regards,

Mark

A few follow on questions.

1) We ran the "show grants" commands as you requested.  Results are at the bottom.  They look OK to me.

2) We believe we already ran add_slave_user.  On /etc/my.cnf, server-id=1 is uncommented on the primary, and server-id=1 is uncommented on the secondary.  Is there something else we should check before running add_slave_user again?  

3) The instructions you referenced also said to run create_slave.sh.  How can we tell if that has already been run?

4) This is our production system.  How do we arrange to have CA support on the phone if we encounter problems we do not know how to handle?

Results of "show grants" commands

xx-APIPROD1

| Grants for repluser@xx-apiprod1.thebank.com |
| GRANT SELECT, RELOAD, LOCK TABLES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repluser'@'xx-apiprod1.thebank.com' IDENTIFIED BY PASSWORD '*D98280F03D0F78162EBDBB9C883FC01395DEA2BF' |
| Grants for repluser@xx-apiprod2.thebank.com |
| GRANT SELECT, RELOAD, LOCK TABLES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repluser'@'xx-apiprod2.thebank.com' IDENTIFIED BY PASSWORD '*D98280F03D0F78162EBDBB9C883FC01395DEA2BF' |

xx-APIPROD2

| Grants for repluser@xx-apiprod1.thebank.com |
| GRANT SELECT, RELOAD, LOCK TABLES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repluser'@'xx-apiprod1.thebank.com' IDENTIFIED BY PASSWORD '*D98280F03D0F78162EBDBB9C883FC01395DEA2BF' |
| Grants for repluser@xx-apiprod2.thebank.com |
| GRANT SELECT, RELOAD, LOCK TABLES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repluser'@'xx-apiprod2.thebank.com' IDENTIFIED BY PASSWORD '*D98280F03D0F78162EBDBB9C883FC01395DEA2BF' |

1. the permission might not be enough, you may want 

grant all on ssg.*

2. if you follow the provided document, and run add_slave_user.sh +  create_slave.sh again on both nodes, it should fix the env problem. (including server-id, it should be different on different node, both set to 1 is wrong.)

3. but the default repl port is 3307, would it be blocked?

you may manually test the connection

run show slave status\G on node1, it will show

mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: <host of node2>
Master_User: <repl user of node2>
Master_Port: <repl port of node2>
...

run the command below on node1 to see if you can login

mysql --host=<host of node2> --user=<repl user of node2> --password=<pass> --port=<repl port> ssg  

4. You can open a support ticket on support.ca.com to get help from CA Support. (You need proper entitlement)

Hi,

Was this problem resolved.

I am seeing same error as you.

The replication was configured and it was working fine for few days but suddenly we are seeing error in th elog.

2017-03-04T17:30:30.552-0600 WARNING 101 com.l7tech.server.cluster.DatabaseReplicationMonitor: 2381: Replication failing for host/database learn1.l7tech.com/ssg: error calculating delay

Let me know what was the resolution for you.

Thank you,

Rajya