One or more of the database servers are not connected or in "Sync" state. What should I do?


  1. Check the database cluster status on all database servers and Callbridge servers. All servers should be Connected and In Sync. 

If one server is not In sync:

In the example below, the server (192.168.3.80) is not in "Sync" state – there are 304 bytes behind. It must lose the connection to the database Primary. 
cms> database cluster status
Status : Enabled

Nodes:
172.16.5.80 : Connected Replica ( In Sync )
192.168.4.81 (me) : Connected Primary
192.168.3.80 : Connected Replica ( 304 bytes behind )
Node in use : 192.168.4.81

When server (192.168.3.80) loses connection, it has no primary.

cms> database cluster status
Status : Enabled

Nodes:
172.16.5.80 : Connected Replica
192.168.4.81 : Disconnected
192.168.3.80 (me) : Connected Replica
Node in use : 172.16.5.80

See What to do if there is no primary database? 

If one server is not connected:

In the example below, the server (192.168.3.80) is not in connected state. The database server (192.168.4.81) and server (192.168.3.80) lose connection to each other, but they can both still see the Primary.

cms> database cluster status
Status : Enabled

Nodes:
172.16.5.80 : Connected Primary
192.168.4.81 (me) : Connected Replica ( In Sync )
192.168.3.80 : Disconnected
Node in use : 172.16.5.80

cms> database cluster status
Status : Enabled

Nodes:
172.16.5.80 : Connected Primary
192.168.4.81 : Disconnected
192.168.3.80 (me) : Connected Replica ( In Sync )
Node in use : 172.16.5.80

As both servers 192.168.4.81 and 192.168.3.80 are able to connect database primary, all functions are working. But it would run into no primary state once the current database primary is not available for any reason. 

  1. Download the log file from the server that is not connected or "In Sync" state and analyze it. (It can be download via SFTP. The file name is “log”.)

Search “sfpool” in the log file. You may find the messages below. This tells us when this database server(192.168.3.80) loses connection to the server 192.168.4.81.

sfpool: Health check 192.168.4.81 primary check failure:
sfpool: health check: up check to 192.168.4.81:5432 failed with Killed

  1. Check network connectivity between the database servers. Database servers use telnet(TCP protocol) to communicate to each other as a way of keep-alive. Check network devices and see if TCP connection is blocked.
  2. Check the server status again. When the network connectivity problem is fixed, all servers should be able to talk to each other and the state for replica servers whould be In Sync as in the example below. 

cms> database cluster status
Status : Enabled

Nodes:
172.16.5.80 : Connected Primary
192.168.4.81 : Connected Replica ( In Sync )
192.168.3.80 (me) : Connected Replica ( In Sync )
Node in use : 172.16.5.80

  1. Check the log file again. When the network connectivity problem is fixed, you should see the message below. 

INFO : Cleared databaseClusterNodeOutOfSync alarm for cluster node 192.168.4.81

  1. If there is no network connectivity issues but it still not in connected or "Sync" state, you may need to appoint a new primary manually.

If none of the above solved the issue, contact Cisco support immediately. Please provide the logs mentioned in the FAQ on database errors or warnings.

Last update:
08-Sep-2020
FAQ ID:
1236