Follow these steps:

1. Check the database cluster status on all database servers and Call bridge server to confirm that there is no primary. 

In the example below, we have three database servers and one call bridge. The Call Bridge sees no primary.

One server (172.16.5.80) is isolated from the other two database servers, which results in it running in replica mode without primary database, as it cannot see any other database server in the cluster.

cms> database cluster status
Status : Enabled
Nodes: 
172.16.5.80 (me) : Connected Replica
192.168.4.81 : Disconnected
192.168.3.80 : Disconnected
Node in use : 172.16.5.80

The other two database servers (192.168.4.81 and 192.168.3.80) are isolated and they both can only see each other.

Database server (192.168.4.81) sees itself as primary and one other database as replica.

cms> database cluster status
Status : Enabled
Nodes: 
172.16.5.80 : Disconnected
192.168.4.81 (me) : Connected Primary
192.168.3.80 : Connected Replica ( In Sync )

Database server (192.168.3.80) sees on other database as primary and itself as replica.

cms> database cluster status
Status : Enabled
Nodes: 
172.16.5.80 : Disconnected
192.168.4.81 : Connected Primary
192.168.3.80 (me) : Connected Replica ( In Sync )
Node in use : 192.168.4.81

The Callbridge (192.168.6.77) can only see database server (172.16.5.80) which is running in replica mode.

cms> database cluster status
Status : Enabled

Nodes: 
172.16.5.80 : Connected Replica
192.168.4.81 : Disconnected
192.168.3.80 : Disconnected
Node in use : 172.16.5.80

Note: This situation is very likely to occur if 3 clustered databases are placed on only 2 locations. See why you need to place databases on different locations.

2. Download the log file from the server that has no database Primary and analyze it. (It can be download via SFTP. The file name is “log”.)
• Search “read only” in the log file, you may find the following messages. This tells us when the database Primary disappeared.

Started Postgres
Postgresql restarted as replica
WARNING : database connection failure (FATAL: the database system is starting up)
WARNING : Database connection read only

• Then search “sfpool” in the log file, look at the sfpool messages immediately before the "read only" message. This tells us when this server was isolated from the other two database servers before it is changed to “read only” mode.

sfpool: health check: up check to 192.168.3.80:5432 failed with Killed
sfpool: health check: up check to 192.168.4.81:5432 failed with Killed 

3. Check network connectivity between those database servers. Database servers use telnet (TCP protocol) to communicate to each other as a way of keep-alive. Check network devices and see if TCP connection is blocked. 

4. After network connectivity problem is fixed, this server should see other database servers and be able to connect to an existing primary or promote itself as a database Primary. See the example below.

cms> database cluster status
Status : Enabled
Nodes: 

172.16.5.80 (me) : Connected Replica ( In Sync )
192.168.4.81 : Connected Primary
192.168.3.80 : Connected Replica ( In Sync )

Node in use : 192.168.4.81

And if you check the log file, the following messages will tell you that the database server is connected to the new primary. 

Starting postgres
Attachment complete
Connected to new primary 192.168.4.81

5. If there is no network connectivity issues, but still no primary presents, you may need to appoint a new primary manually. 

To appoint a new primary, please refer to https://meeting-infohub.cisco.com/faq/content/11/75/en/how-do-i-move-the-master-database.html

If none of the above solved the issue, contact Cisco support immediately. 

Last update:

08-Sep-2020

FAQ ID:

1213