Layer7 API Management

Hi Team,

we are getting gateway node startup errors, please find below errors. 

2018-10-15T04:13:19.996+0800 WARNING 1 STDERR: com.l7tech.server.LifecycleException: Hibernate operation: Could not execute JDBC batch update; SQL [update cluster_info set address=?, avgload=?, uptime=?, esm_address=?, statustimestamp=?, mac=?, multicast_address=?, name=? where nodeid=?]; Lock wait timeout exceeded; try restarting transaction; nested exception is java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transaction 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.cluster.ClusterBootProcess.start(Unknown Source) 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.BootProcess.start(Unknown Source) 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.boot.GatewayBoot.f(Unknown Source) 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.boot.GatewayBoot.start(Unknown Source) 2018-10-15T04:13:19.997+0800 WARNING 1 STDERR: at com.l7tech.server.boot.GatewayBoot.runUntilShutdown(Unknown Source) 2018-10-15T04:13:19.998+0800 WARNING 1 STDERR: at com.l7tech.server.boot.GatewayMain.main(Unknown Source) 2018-10-15T04:13:19.998+0800 WARNING 1 STDERR: Caused by: org.springframework.dao.CannotAcquireLockException: Hibernate operation: Could not execute JDBC batch update; SQL [update cluster_info set address=?, avgload=?, uptime=?, esm_address=?, statustimestamp=?, mac=?, multicast_address=?, name=? where nodeid=?]; Lock wait timeout exceeded; try restarting transaction; nested exception is java.sql.BatchUpdateException: Lock wait timeout exceeded; try restarting transaction

Please help me to fix ASAP.

Hi Mark,

Thanks for quick reply, we have rebooted server but no luck, no other process is locking DB table.

there is something else. yes, DB and ssg process on same host.

Uday,

You can check if there is a deadlock created(If run within the lock wait time out period) on the table by executing 

mysql -e "SHOW ENGINE INNODB STATUS\G;"

If there are any, it should show something like below.

mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 376, 1 row lock(s)
MySQL thread id 493227, OS thread handle 0x7f3f49cb3700, query id 46546607 localhost root statistics
select * from test.testlock where a = 1
------- TRX HAS BEEN WAITING 16 SEC FOR THIS LOCK TO BE GRANTED:

SHOW PROCESSLIST; would show the query as "Searching rows for update"

Hi Anand,

No luck .. 

A case was opened for this issue and at some point the system were recovered so root cause was not determined.

Sincerely,

Stephen Hughes

Broadcom Support