Alerts
To quickly view any notification alerts, use the Notifications label on the header bar of the Command Center dashboard.
When you open the Notifications window, unread notifications are highlighted in light blue. The alerts are ordered by time and are kept for 7 days. When you close the Notifications window, all alerts are considered read.
To customize the alert notifications, go to the Admin> Alerts page, where an administrator can set up alert rules. The rules help detect and respond to events that occur in the Greenplum Database system and in currently executing database queries. When a rule is matched, Command Center logs a record.
You can set up email alerts by configuring an SMTP server in Greenplum Database or in Command Center. Additionally, you can create a send_alert.sh
shell script to forward alerts to other destinations, such as an SMS gateway or a Slack channel. If the script is present, Command Center runs it whenever an alert is raised.
Command Center creates the gpmetrics
schema in the gpperfmon database to store both rules and log records. See gpmetrics Schema Reference for information about the gpcc_alert_rule
and gpcc_alert_log
tables in the gpmetrics
schema.
This topic contains the following subtopics:
Configuring Alert Rules
Click EDIT to manage alert event rules. To enable an alert rule, enter any data required in the fields and check the box. Uncheck the box to disable the rule. Click SAVE when you have finished making changes to the alert configuration.
Cluster Level
- Command Center is running on the master host, the master host is up, but the database is down or cannot be reached.
- Command Center is running on the standby master host, the standby master host is up, but the master host is down or cannot be reached, or the database is down or cannot be reached.
- Command Center checks three times before raising the alert. If the host where Command Center should be running is down, no alert is raised.
Master Level
Segment Level
100 - sum(<available disk space>) / sum(<total disk space>) * 100A disk space alert is raised no more than once every 24 hours.
Query Level
Configuring Alert Email
Command Center requires an SMTP server to send alert emails.
Command Center first attempts an encrypted TLS connection and then falls back to an unencrypted connection if TLS is not supported. The SMTP server must support one of the following authentication methods: NONE, PLAIN, LOGIN, or CRAM-MD5. Command Center will use the most secure of these methods the SMTP server allows.
Configuring email With Command Center
Click EDIT in the Manage email configuration panel.
The alert email configuration is set with the following Greenplum Database server configuration parameters:
smtp.example.com:465
gpcc-alerts@example.com
X
on the address.From:
address in the alert email. Example: do-not-reply@example.com
.
If you leave this field blank, Command Center uses the default value, noreply-gpcc-alerts@pivotal.io
.
When you click SAVE, Command Center sends a test email to the addresses in the Send emails to field. The email contains a list of the currently configured alert rules. If there is an error in the SMTP server or username/password configuration and the email cannot be sent, Command Center displays an error message.
Configuring email for Greenplum Database
The following server configuration parameters are used to configure SMTP email for Greenplum Database.
smtp.example.com:465
gpcc-alerts@example.com
noreply-gpcc-alerts@example.com
gpcc-admin@example.com;gpdb-admin@example.com
Command Center uses the gp_email_smtp_server
, gp_email_smtp_userid
, and gp_email_smtp_password
parameters if they are set. It ignores the remaining parameters.
You can check the current value of a configuration parameter by running the gpconfig -s
command on the master host, for example:
$ gpconfig -s gp_email_smtp_server
Use the gpconfig -c
option to set the values of server configuration parameters, for example:
$ gpconfig -c gp_email_smtp_server -v "smtp.example.com:465"
$ gpconfig -c gp_email_smtp_userid -v "gpcc-alerts@example.com"
$ gpconfig -c gp_email_smtp_password -v "changeme"
$ gpconfig -c gp_email_from -v "gpcc-alerts@example.com"
$ gpconfig -c gp_email_to -v "gpcc-admin@example.com;gpdb-admin@example.com"
Run gpstop -u
to reload the configuration files after changing these configuration parameters.
Creating a Send Alert Script
The send alert script is a shell script that you can use to send Command Center alerts to destinations such as SMS gateways, pagers, team collaboration tools like Slack, chat servers, archive files, alternative email servers, and so on. You can use the send alert script in addition to sending email from Command Center, or as an alternative to sending alert emails from Command Center.
Command Center looks for the script $MASTER_DATA_DIRECTORY/gpmetrics/send_alert.sh
on the host where Command Center is running—either the master host or standby host. If the file exists and is executable by the gpadmin user, Command Center executes the script. The following variables are set on the command line when the script runs.
Variable | Description |
---|---|
LINK | URL of the Greenplum Command Center web server. |
QUERYID | ID of the query, if the alert was triggered by a query. |
SERVERNAME | Name of the Greenplum Command Center server. |
QUERYTEXT | The text of the query, if the alert was triggered by a query. |
ACTIVERULENAME | Current text of the rule, with user-specified values included. |
LOGID | Value of this alert’s id column in the gpmetrics.gpcc_alert_log table. |
RULEDESCRIPTION | Text of the rule, including user-specified values, at the time the alert was raised. |
ALERTDATE | Date the alert was raised. |
ALERTTIME | Time the alert was raised. |
SUBJECT | Subject line for email. |
An example script that you can customize is provided at $GPCC_HOME/alert-email/send_alert.sh.sample
. The example formats the alert as HTML email text and pipes it through the Linux mail
command.
To set up a send alert script:
Copy the
$GPCC_HOME/alert-email/send_alert.sh.sample
file to$MASTER_DATA_DIRECTORY/gpmetrics/send_alert.sh
.Customize the script with code to format and deliver the alert to your desired destination.
Run
gpcc start
to restart Command Center and enable the script.
SNMP Alert Script Example
SNMP support is discontinued in Greenplum Database 6, but you can use a Command Center alert script to send alerts to an SNMP manager.
This example alert script sends Command Center alerts as SNMP notifications in a format similar to Greenplum Database 5 SNMP notifications, using the SNMP OIDs for Greenplum Database as arguments.
This is the OID hierarchy structure for Greenplum Database:
iso(1)
identified-organization(3)
dod(6)
internet(1)
private(4)
enterprises(1)
gpdbMIB(31327)
gpdbObjects(1)
gpdbAlertMsg(1)
The following table lists the Greenplum Database OIDs:
Name | Address | Type: Description |
---|---|---|
gpdbAlertMsg | 1.3.6.1.4.1.31327.1.1 | string: alert message text |
gpdbAlertSeverity | 1.3.6.1.4.1.31327.1.2 | integer: severity level |
gpdbAlertSqlState | 1.3.6.1.4.1.31327.1.3 | string: SQL standard error codes |
gpdbAlertDetail | 1.3.6.1.4.1.31327.1.4 | string: detailed alert message text |
gpdbAlertSqlStmt | 1.3.6.1.4.1.31327.1.5 | string: SQL statement generating this alert, if applicable |
gpdbAlertSystemName | 1.3.6.1.4.1.31327.1.6 | string: hostname |
Command Center sets environment variables that can be mapped to the gpdbAlertMsg, gpdbAlertSqlStmt, and gpdbAlertSystemName OIDs. For the other OIDs, the script sends an empty string or 0, depending on data type.
Prerequisites
The SNMP service must be configured and enabled on the host where Command Center is running. If the snmpd
file is not present in the /usr/sbin
directory, then SNMP is not installed on the system.
To install SNMP on Red Hat and CentOS, use yum
to install the net-snmp
, net-snmp-libs
, and net-snmp-utils
packages. Configure SNMP following the operating system documentation. This example script specifies the SNMP community public
, but you can set a different community string with the rocommunity
parameter in the /etc/snmp/snmpd.conf
file.
Start the snmpd
daemon with the systemctl
command:
$ sudo systemctl enable snmpd.service
Verify the system snmpd
daemon is running with this command:
$ sudo snmpwalk -v 1 -c public localhost .1.3.6.1.2.1.1.1.0
If the daemon is running, the snmpwalk
command output is similar to the following:
SNMPv2-MIB::sysDescr.0 = STRING: Linux gp-mdw 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64
Setting up SNMP Notifications
As the
gpadmin
user, create the file$MASTER_DATA_DIRECTORY/gpmetrics/send_alert.sh
with the following contents:RUNNINT_TIME=`psql -c 'select extract(epoch from (current_timestamp - pg_postmaster_start_time())) * 100;'|sed -n '3p'|awk '{print $1}'` HOST=`hostname` snmptrap -v 2c -c public \ {##trapserver##} \ "${RUNNINT_TIME}" \ "enterprises.31327.5.0.1" \ SNMPv2-SMI::enterprises.31327.1.1 s "${RULEDESCRIPTION}" \ SNMPv2-SMI::enterprises.31327.1.2 i 0 \ SNMPv2-SMI::enterprises.31327.1.3 s "" \ SNMPv2-SMI::enterprises.31327.1.4 s "" \ SNMPv2-SMI::enterprises.31327.1.5 s "${QUERYTEXT}" \ SNMPv2-SMI::enterprises.31327.1.6 s "${HOST}"
Replace
{##trapserver##}
with the name of the host runningsnmpd
.Make the script executable.
$ chmod +x $MASTER_DATA_DIRECTORY/gpmetrics/send_alert.sh
Restart Command Center to enable the script.
$ gpcc start
Testing the SNMP Alert Script
To test the script, you can use the snmptrapd
trap receiver.
As root, edit the
/etc/snmp/snmptrap.conf
file and add the following line to allow the daemon to receive unauthenticated notifications:disableAuthorization yes
Start the
snmptrapd
daemon with this command:# /usr/sbin/snmptrapd -m ALL -Lf ~/snmptest.log
Check the /home/root/snmptest.log
file for notifications.