Pivotal Greenplum® Command Center 4.7.0 Release Notes
Last update 12/23/2019
This document contains release information about Pivotal Greenplum Command Center 4.7.0. Greenplum Command Center 4.7.0 adds new features to Greenplum Command Center and resolves some issues.
See Enhancements and Changes in Greenplum Command Center 4.7.0 for information about new features and changes in this Command Center release.
Greenplum Command Center 4.7.0 is compatible with the following platforms.
- Pivotal Greenplum Database 5.19.0 and higher
- Red Hat Enterprise Linux 6.x1 and 7.x
- CentOS 6.x1 and 7.x
- SUSE Enterprise Linux 11 SP42
1If you use resource groups and workload management on Red Hat or CentOS 6.x, upgrade your kernel to 2.6.32-696 or higher to benefit from improvements in the Linux cgroups service.
2Greenplum Command Center workload management features are not supported on SUSE Enterprise Linux 11.
Greenplum Command Center 4.7.0 contains the following new changes and features.
Command Center Installation New Features and Changes
gpccinstallinstallation utility now writes details of the installation to the installation log file,
gpccinstall.<timestamp>.log. Previously, the log contained a success message or any errors that prevented a successful message. The log now contains the values for the parameters used during the Command Center installation.
gpccinstallutility must be run by the gpadmin user and the installation directory must be writable by the gpadmin user. The option to install Command Center as root and change ownership of the installed files to gpadmin is not supported because the root user cannot connect to Greenplum Database.
gpccinstall -autooption performs a non-interactive installation, using default values for all of the installation parameters.
Added the Korean language option for the Command Center User Interface.
It is now recommended to run Command Center on the master host, instead of the standby master host. Saving plan node history, a new feature in this Comand Center release, can generate a large number of messages between the agent running on the master host and the Command Center backend. Running Command Center on the master host keeps this traffic off of the LAN.
gpmetrics Schema Changes
- When real-time metrics collection is enabled, Command Center saves execution plan node details to the
gpmetrics.gpcc_plannode_historytable in the gpperfmon database. Details are only saved for queries that execute for at least 10 seconds. The data in this table enable you to view visual query execution plan details when you select a query in the History view.
Query Monitor and History Changes
Plannerfield is added to the Query Monitor Details view to identify the planner used to plan the query, either GPORCA or Legacy.
The detail for a query in the History view includes a visual query execution view that shows the actual rows returned for each plan node. History
The visual plan for a query in the Query Monitor shows the Estimated Progress for the query, a calculated percentage using the query plan’s cost estimate for each plan node and the actual rows vs. estimated rows for each node.
When sending alert emails, Command Center can authenticate with the SMTP server using NONE, PLAIN, LOGIN, or CRAM-MD5 authentication.
The Username and Password fields in the Manage email configuration section of the Alerts page are now optional. These fields are only needed if the SMTP server requires them for authentication.
- Every hour, Command Center logs the number of queries that were not saved to the gpmetrics history in the previous hour because they completed in less time than than the threshhold time specified on the Admin> Settings page. The message is logged in the
logs/webserver.logfile in the Command Center installation directory.
165236914 - The
gpcccommand-line utility did not work with the
-Woption. This has been fixed. The
gpcccommand will use the gpmon password entered by the user when the
-Woption is supplied. If the
-Woption is not supplied,
gpccwill use the
PGPASSWORDenvironment variable, if set, or the
.pgpassfile in the gpadmin user’s home directory.
165111508 - LDAP authentication fails because Command Center improperly encodes the username in the Greenplum connection string. This is resolved. Command Center correctly escapes the username in the connection string.
164620455 - The Command Center agent memory usage and CPU usage grew very high when executing a query with an inner query. The inner query generated many small queries, sending a large number of packets to the metrics collector. This is fixed. Inner queries are marked so that metrics are not collected for them.
164818798 - When Workload Manager terminates an idle connection, a FATAL error and stack trace were logged. Now Workload Manager logs a WARNING message with no stack trace when an idle connection is terminated.
164263538, 164718926 - Command Center does not show query information for a COPY command, such as
COPY <table> FROM STDIN;. The Command Center Query Monitor now displays available information for some
COPYcommand run by a non-superuser role can be controlled by a resource queue if the resource queue’s
ACTIVE_STATEMENTSparameter specifies a maximum limit for the number of queries that can be executed by roles assigned to the queue. (Greenplum Database does not assign cost or memory values to
COPYcommands, so a
COPYcommand with no
ACTIVE_STATEMENTSparameter is not managed by a resource queue.) In Command Center,
COPYcommands managed by resource queues will now appear in the Query Monitor with available activity information. Command Center caches roles’ resource queue names and, for these commands, the resource queue name is displayed from the cache. If a role has been recently assigned a different resource queue, the cache may be out of date. See Known Issues for more information.
164346696 - Command Center does not display more than one filespace per host. This has been fixed in Command Center.
165589279 - The Command Center Dashboard displayed UTC time when the Europe/Moscow timezone was set in the OS and Greenplum Database. The root cause of this issue is that Greenplum Database 5 does not handle the
MSKtime zone properly. This is fixed in Command Center and issue 7543 has been opened for Greenplum Database.
Greenplum Command Center 4.6.0 contains the following new features:
Display Language Selection
The Greenplum Command Center user interface has been updated with localization support. At installion time, you can choose the language to use in the Command Center user interface. Command Center currently supports English (default) and Chinese languages.
gpmetrics History Schema (Beta)
The Command Center history views currently draw data from the Greenplum Database gpperfmon database, which has a data collection process that is separate from the Command Center real-time metrics collection system.
In this release, you can enable saving data from the real-time metrics collection system to history tables in the gpmetrics schema. When you enable this feature, Command Center uses the gpmetrics history tables to display history data in the user interface. This is a new feature, currently considered experimental. In a future release, the gpmetrics tables will be the only source for history data and Command Center will no longer depend upon the gpperfmon database.
To enable gpmetrics history, visit the Admin> Settings view in Command Center. In addition to enabling history collection, you can set the minimum run-time for queries to be saved in history.
See the gpmetrics Schema Reference for information about the tables in the gpmetrics schema.
Greenplum Command Center 4.5.0 contains the following new features:
On the new Admin> Alerts page, Command Center users with superuser permission can configure alert rules. When an alert rule is matched, a record is logged in the
gpmetrics.gpcc_alert_log table and, optionally, an email is sent to a list of addresses you specify. You can also write a custom shell script to send alerts to other messaging systems.
Alert rules can detect the following events:
- Segment failure
- Out of memory errors
- Average memory usage percentage of segment hosts exceeds a specified percentage for a specified number of minutes
- Memory usage on the master host exceeds a specified percentage for a specified number of minutes
- Total disk space used on all hosts exceeds a specified percentage
- Number of Greenplum Database connections exceeds a specified percentage
- Average CPU usage of segment hosts exceeds a specified percentage for a specified number of minutes
- Spill files for an active query exceed a specified number of gigabytes
- Runtime for a query exceeds a specified number of minutes
- A query is blocked for more than a specified number of minutes
Query Text Download for Long Queries
On the Query Details page, if the query text is longer than 100K characters, Command Center only shows the first 100K characters and, when you click COPY, copies only the first 100K characters to the clipboard. When the query text is longer than 100K characters, Command Center adds a button, Retrieve full query text, which you can click to download a text file containing the entire text of the query. The file is available to download for 24 hours, or until the query is saved to history, when history collection is enabled.
161436520 - Panels displaying help text can now be scrolled vertically when the text does not fit in the window.
158278655 - The displayed metrics for a query could be inaccurate if Command Center received the query “done” status before the final metrics arrived. Now the final metrics are updated even when they arrive after the “done” status.
161074984 - Repeated assertion errors logged in the
agent.logfile caused log files to grow quickly and consume too much disk space. This is fixed. Each type of assertion error will be logged no more than once every 10 minutes.
162114428 - When a user visits Command Center with a browser, the Command Center web server (gpccws process) establishes a websocket connection and runs two goroutines to service the websocket. When the websocket closes, one of the goroutines does not exit. After many connections the gpccws process can occupy gigabytes of memory. This memory leak is fixed.
- The Greenplum Database metrics collector extension is now enabled only when Command Center agents are running. Previously, if an agent process terminated, the metrics collector continued to collect and send data.
Greenplum Command Center 4.4.2 contains the following resolved issues:
160195564 - When Command Center is restarted after a user has submitted a query, and the user submits another query in the same session, the database name and user name are missing from the second query in the Command Center interface. This is fixed.
161077294 - The Authentication view now prevents the user from saving an undefined entry to the
161274227 - The Command Center installer,
gpccinstall, no longer prints errors to the
pg_loglog file about missing
161273865 - The Query Monitor no longer shows runtimes flashing to 0 seconds when a query is cancelled through the user interface.
The Admin> Authorization view allows entering host names in the Address field.
On the visual query plan, a data motion showed a finished status when the send data operation completed. This is changed so that the node completes only after the corresponding data receive operation has also completed.
Enabled cross-site request forgery prevention during login.
Resolved Issues in Greenplum Command Center 4.4.1
160679694 - If Command Center is restarted while on the Query Monitor view and the browser is then refreshed, an extra web socket connection is created. This is fixed.
MPP-29539 - Command Center agent (ccagent) logging is disabled for a known issue with PL/pgSQL queries.
Greenplum Command Center 4.4.0 contains the following features and enhancements.
Command Center has a new user interface to assist administrators in enabling resource groups in Greenplum Database, importing existing resource queues to resource groups, and enabling workload management with Command Center. The option to import resource queues to resource groups is presented if no resource groups have been created (other than
admin_group) and Greenplum Database has resource queues to convert (other than
pg_default). Once the administrator has imported resource queues, or chosen to skip importing resource queues, the option to import queues is no longer presented.
The resource group list on the Workload Management view has a new column to show the minimum (fixed) amount of memory that will be allocated to a query for each resource group. This value is recalculated when you enter new values while editing resource groups.
Administrators can now define resource group assignment rules and idle session kill rules with an interactive interface. It is no longer necessary to edit the JSON document for workload management rules. The JSON text field is removed.
The Workload Management view changed to a light theme.
Command Center users with the Self Only permission level can:
- see all queries on the query monitor, including queries owned by other users
- cancel their own queries
- access the query details view for their own queries
- hover on a query to see query text for their own queries
- hover on locking/blocking queries and access details of locking/blocking queries that do not belong to other users
Users with Self Only permission level cannot see query text or access the details views of others’ queries.
The Greenplum Database roles
gpcc_operator_basicare created during Command Center installation if they do not already exist.
- Fixed a bug where the database name and role name were missing from query details when queries are executed in a session after restarting Command Center.
Greenplum Command Center 4.3.0 contains the following enhancements.
Resource Group Management
The Admin>Workload Mgmt view has a new user interface you can use to add and remove resource groups and to change the Concurrency, CPU %, and Memory % attributes of resource groups.
Resource Group Role Assignments
The Admin>Workload Mgmt view has a new user interface you can use to view and change Greenplum Database roles’ default resource groups.
Details Added to Visual Query Plan Steps
The metrics collector extension in Greenplum Database release 5.9 is updated to submit additional information about each step in the query plan to the Command Center backend. Command Center displays this information in the visual query plan when you expand a step in the query plan. The new information displayed depends on the operation the step performs and includes details such as hash key, merge key, join condition, or filter condition. Previously, you could only see this information by generating the textual query plan.
Greenplum Command Center 4.2.0 contains the following enhancements.
Visual Query Plan
The Command Center Query Details view now includes a visual query plan.
Idle Session Kill Rules
Idle session kill rules can include optional
- The value of the
exemptedRolesparameter is a list of Greenplum Database roles that are exempted from the rule. The list can include Posix regular expressions to match Greenplum Database role names.
- The value of the
messageparameter is a string to include in the message that is displayed when a session is killed.
Command Center Can Run on the Master Host or Standby Master Host
The Greenplum Command Center web server and backend may now be executed on the master host or on the standby master host. Running GPCC on the standby master host is recommended to avoid adding load to the master server, but it is no longer a requirement. After the GPCC software is installed, log in to the host where you want to run GPCC, source the
gpcc_path.sh file in the GPCC installation directory, and run the
gpcc start command.
Greenplum Command Center 4.1.0 contains the following enhancements.
- Command Center administrators can set permission levels for Command Center users. Permissions are enforced as described in the documentation.
On the Query Detail view, clicking Copy in the query text or query plan panel copies the text in the panel to the clipboard.
A help icon and in-app help have been added on the Query Monitor and Query Detail views.
The following workload management features, improvements, and bug fixes have been added in the workload management extension included with Greenplum Database 5.8.0.
On the Admin>Workload Mgmt view, you can add idle session kill rules for each resource group. When you add these rules, the Greenplum Database workload management extension kills a session after it has been idle for the number of seconds you specify. See Workload Management for syntax and examples.
Optimizations have been implemented to reduce the impact on Greenplum Database when the workload management extension is disabled.
The workload management extension takes advantage of resource group name-to-id caching added in Greenplum Database.
Fixed a bug in the workload management extension that caused errors to print to the
psqlprompt, even when the extension was disabled.
Greenplum Command Center 4.1.0 contains the following bug fixes.
- In the History Detail view, when either of the Disk R or Disk W metrics is 0, both are reported to be 0. This is fixed.
- When using the Kerberos gpmon-only authentication mode, generating an explain plan failed. This is fixed.
- The Command Center agent failed with the message “Error: can’t find gpcc.query_metrics_port, metrics_collector is not correctly installed.” This occurs when running Command Center on a Greenplum Database system that was upgraded from an earlier Greenplum Database 5.x release. The
gp_wlmextensions are installed with the upgrade, but the upgrade process does not perform the required configuration changes in the
postgresql.confconfiguration file. The Command Center installation instructions now include steps to manually configure and restart an upgraded Greenplum Database system.
- When Command Center starts, an error message is written in the Greenplum Database log file: “function gpcc_schema.read_pghba(unknown) does not exist.” This is fixed.
- A change to time zone handling in Greenplum Database 5.7 can cause Greenplum Command Center 4.0.0 to display an incorrect time if the master host operating system time and Greenplum Database use different time zones. Now Command Center times are displayed using the time zone of the Greenplum Database master host operating system. The current time, last sync time, and timestamps in alert logs, cluster metrics, and query history are all displayed using the master host’s system time zone.
- On the Admin> Authorization view, when the authorization method is
gssand the options field contains text, changing the method to
trustdoes not clear the options field and it is not possible to save changes to the
- The Command Center web server,
gpccws, spawns ssh processes but does not reap them in a timely manner, leading to many zombie processes. This is fixed.
- On the Admin> Authentication view, if the number of users listed in the user column is longer than can be displayed, Command Center truncates the list and adds ellipsis (
...) to the end. Only administrators can view the complete value, by editing the field. Now any user with access to the view can see the full list of users by hovering over the field.
Command Center Installation Changes
Greenplum Command Center 4.x software, unlike previous releases, is installed on every host in the Greenplum Database cluster. The Command Center web server and backend run on the standby master, if your Greenplum Database cluster has a standby master. If there is no standby master, Command Center runs on the master host instead.
To modify the Command Center installation—for example to enable or disable SSL or install Command Center on new or replaced hosts—just re-execute the installer and restart Command Center. It is not necessary to uninstall Command Center before reinstalling.
There is one Command Center installation per Greenplum Database cluster. It is no longer necessary to create Command Center instances after installing the software.
gpcmdr command-line utility is replaced with the new
gpcc utility. Use the
gpcc utility to start and stop Command Center and metrics collection agents, check Command Center status, and enable or disable Kerberos authentication.
In previous releases, the gpmon role required only local connections to databases on the Greenplum master host. In Greenplum Command Center 4.x, the gpmon user must be able to connect to databases from the host running the Command Center web server and backend, which requires adding a host entry to the
pg_hba.conf authentication configuration file.
Real-time Query Metrics
Greenplum Command Center 4.0 introduces real-time query metrics for Pivotal Greenplum Database 5.7 and above. This new feature combines the following new features in Greenplum Database and Greenplum Command Center:
- Greenplum Database saves query execution metrics in shared memory while queries execute.
- A new Greenplum Database metrics collection extension, included with Pivotal Greenplum Database, emits the saved metrics as UDP datagrams.
- A new Greenplum Command Center metrics collection agent running on each Greenplum Database host receives the datagrams and posts metrics to the Greenplum Command Center backend. The Command Center backend starts and manages the metrics collection agents.
The Command Center Query monitor view updates in real time so you can see queries that are waiting to execute and the current status and resource usage for queries that are running.
Metrics collection now includes lock and spill file information. On the Query Monitor, you can see which queries are blocked and which queries hold the locks blocking them. The Query Monitor shows the total size of spill files created on all segments for each query.
Installing the gpperfmon database remains a prerequisite for Command Center. The gpperfmon database is the source for query history displayed in the Command Center user interface. The new real-time metrics are not persisted and are not directly related to the metrics collected and persisted in the gpperfmon database.
Workload management is now an integrated Command Center feature rather than a separate product. Workload management is available in Command Center only after resource groups have been enabled in Greenplum Database by changing the
gp_resource_manager server configuration parameter from
'group' and enabling Linux control groups (cgroups).
In Command Center 4.0, workload management allows you to assign transactions to Greenplum Database resource groups at execution time by evaluating the current database role and query tags against workload assignment filters you define in Command Center. Query tags are user-defined
name=value parameters that you define in the
gpcc.query_tags database session parameter. You can define multiple query tags separated by semicolons. Set query tags in a Greenplum Database session either as a parameter in the database connection URL or by executing
SET gpcc.query_tags TO '<tag1>=<val1>;<tag2>=<val2>;...' in the database session.
When a transaction is about to execute, the current database role and query tags are compared to the workload assignment filters that you have created in Command Center. If a match is found, the transaction is assigned to a resource group according to the workload management filter. Otherwise, the transaction is assigned to the database user’s resource group, which is the default behavior when Command Center workload management is not enabled.
Workload management uses the
gp_wlm database extension included with Pivotal Greenplum Database.
Some features available in previous Greenplum Command Center releases have been removed or are not yet implemented in Command Center 4.x.
- The ability for a Command Center admin to post a message to the Query Monitor view is not yet implemented.
- The multi-cluster view has been removed.
The following are known issues in the current Greenplum Command Center release.
Command Center Installer Truncates pg_hba.conf LDAP Entries
When the Command Center
gpccinstall installer updates the
pg_hba.conf file it truncates LDAP information on entries that use the
ldap authentication method. The truncated information must be reinserted manually after the Command Center installation is complete. This issue exists in Command Center versions 4.7 through 4.9, 6.0, and 6.1.
Failure to Upgrade Plan Node History Table
[MPP-30259] The definition of the
gpmetrics.gpcc_plannode_history table changed in Command Center 4.7. When upgrading from Command Center 4.4, 4.5, or 4.6 to Command Center 4.7 or 4.8, the installer does not update the table definition. This causes a failure to harvest data into the
gpcc_plannode_history table. This issue is fixed in Command Center 4.9.0. Upgrading to Command 4.9.0 correctly updates the table definition.
Failure to Auto-Create Monthly Partitions in gpmetrics Schema
In time zones with daylight savings time (DST) ending in the month of October, Greenplum Command Center fails to create new October partitions for tables in the gpmetrics schema because it specifies the same start and end date for the partitions. The error does not occur in time zones that do not have DST, or in time zones with DST ending in November.
Manually create partitions for October 2019.
Log in to the Greenplum master server as the gpadmin user and connect to the gpperfmon database as the gpmon user:
psql -U gpmon gpperfmon
Enter the following commands to create the October 2019 partitions:
ALTER TABLE gpmetrics.gpcc_disk_history ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE WITH (APPENDONLY=true, ORIENTATION=column, COMPRESSTYPE=RLE_TYPE, COMPRESSLEVEL=2); ALTER TABLE gpmetrics.gpcc_system_history ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE WITH (APPENDONLY=true, ORIENTATION=column, COMPRESSTYPE=RLE_TYPE, COMPRESSLEVEL=2); ALTER TABLE gpmetrics.gpcc_queries_history ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE WITH (APPENDONLY=true, ORIENTATION=column, COMPRESSTYPE=RLE_TYPE, COMPRESSLEVEL=2); ALTER TABLE gpmetrics.gpcc_plannode_history ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE WITH (APPENDONLY=true, ORIENTATION=column, COMPRESSTYPE=RLE_TYPE, COMPRESSLEVEL=2); ALTER TABLE gpmetrics.gpcc_database_history ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE WITH (APPENDONLY=true, ORIENTATION=column, COMPRESSTYPE=RLE_TYPE, COMPRESSLEVEL=2); ALTER TABLE gpmetrics.gpcc_alert_log ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE WITH (APPENDONLY=true, ORIENTATION=column, COMPRESSTYPE=RLE_TYPE, COMPRESSLEVEL=2); ALTER TABLE gpmetrics.gpcc_pg_log_history SPLIT DEFAULT PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE INTO(PARTITION t2019Oct, DEFAULT PARTITION);
Query Monitor is Empty When Using HTTP Proxy
When accessing the Command Center frontend or backend with an HTTP proxy in place, no query activity is displayed in the Query Monitor. To work around this issue, the proxy must be disabled for Command Center client browser connections and for
ccagent processes connecting to the Command Center backend from the Greenplum master and segment hosts. See Using an HTTP Proxy Server with Command Center for more information about this issue.
The gpcc Command Requires the MASTER_DATA_DIRECTORY Environment Variable to be Set
 When starting Command Center with the
gpcc command, if the
MASTER_DATA_DIRECTORY environment variable is not set, Command Center creates the
gpmetrics directory in the current directory. This can cause problems if Command Center is started later with the
MASTER_DATA_DIRECTORY environment variable set.
A Large Memory Leak Occurs Over Time When Certain Alerts are Enabled
[MPP-29942] A deadlock can occur when reading incoming metrics if any of the following alerts are enabled:
- Average memory (segment hosts) exceeds % for min
- Memory (master) exceeds % for min
- Average CPU (segment hosts) exceeds % for min
- CPU (master) exceeds % for min
The deadlock causes incoming metrics to be blocked and accumulated in memory.
To work around this issue, disable all of the above alerts and restart Command Center.
This issue exists in Greenplum Command Center 4.6.1 and 4.7.0.
Unable to View Real Time Queries After Upgrading From a Previous Command Center Release
If you install a new version of Greenplum Command Center using the same port number as the previous version, and you use the Chrome web browser, you may be unable to view real-time queries until after you clear the browser’s cache. See the note in Connecting to the Command Center Console for steps to clear the browser cache.
Resource Queue Name Incorrect for Some Queries
To display the resource queue name for queries in the Query Monitor, Command Center caches user IDs with resource queue names. The cache is updated when the Command Center backend receives a planned query from the metrics collector. Some utility commands are not planned, such as
COPY commands that have no
SELECT clause. These commands use a resource queue slot, but are not passed through an optimizer. If a user is assigned a different resource queue and then executes an unplanned command, the Command Center Query Monitor will display the resource queue name that was previously cached.