Pivotal Greenplum® Command Center 4.6.1 Release Notes

Updated 9/30/2019

About This Release

This document contains release information about Pivotal Greenplum Command Center 4.6.1. Greenplum Command Center 4.6.1 resolves some issues found in Greenplum Command Center 4.6.0.

See Enhancements and Changes in Greenplum Command Center 4.6.1 for information about new features and changes in this Command Center release.

Supported Platforms

Greenplum Command Center 4.6.1 is compatible with the following platforms.

  • Pivotal Greenplum Database 5.17.x and 5.18.x
  • Red Hat Enterprise Linux 6.x1 and 7.x
  • CentOS 6.x2 and 7.x
  • SUSE Enterprise Linux 11 SP42

1If you use resource groups and workload management on Red Hat or CentOS 6.x, upgrade your kernel to 2.6.32-696 or higher to benefit from improvements in the Linux cgroups service.

2Greenplum Command Center workload management features are not supported on SUSE Enterprise Linux 11.

Enhancements and Changes in Greenplum Command Center 4.6.1

Greenplum Command Center 4.6.1 resolves the following issues:

MPP-29782 — When a Greenplum Database System is configured with multiple filespaces, Command Center only displays one of them in the File Storage view. This is fixed.

164162623 — When installing Greenplum Command Center 4.6.0, the installer printed an erroneous message that the newly installed Command Center is not compatible with Greenplum Database 5.17. The message has been removed.

164294306 — When you click a Query Id on the Command Center History page, the Query Detail page is sometimes displayed with no query data. This has been fixed.

Enhancements and Changes in Greenplum Command Center 4.6.0

Greenplum Command Center 4.6.0 contains the following new features:

Display Language Selection

The Greenplum Command Center user interface has been updated with localization support. At installion time, you can choose the language to use in the Command Center user interface. Command Center currently supports English (default) and Chinese languages.

gpmetrics History Schema (Beta)

The Command Center history views currently draw data from the Greenplum Database gpperfmon database, which has a data collection process that is separate from the Command Center real-time metrics collection system.

In this release, you can enable saving data from the real-time metrics collection system to history tables in the gpmetrics schema. When you enable this feature, Command Center uses the gpmetrics history tables to display history data in the user interface. This is a Beta feature. In a future release, the gpmetrics tables will be the only source for history data and Command Center will no longer depend upon the gpperfmon database.

To enable gpmetrics history, visit the Admin> Settings view in Command Center. In addition to enabling history collection, you can set the minimum run-time for queries to be saved in history.

See the gpmetrics Schema Reference for information about the tables in the gpmetrics schema.

Enhancements and Changes in Greenplum Command Center 4.5.0

Greenplum Command Center 4.5.0 contains the following new features:

Alert Rules

On the new Admin> Alerts page, Command Center users with superuser permission can configure alert rules. When an alert rule is matched, a record is logged in the gpmetrics.gpcc_alert_log table and, optionally, an email is sent to a list of addresses you specify. You can also write a custom shell script to send alerts to other messaging systems.

Alert rules can detect the following events:

  • Segment failure
  • Out of memory errors
  • Average memory usage percentage of segment hosts exceeds a specified percentage for a specified number of minutes
  • Memory usage on the master host exceeds a specified percentage for a specified number of minutes
  • Total disk space used on all hosts exceeds a specified percentage
  • Number of Greenplum Database connections exceeds a specified percentage
  • Average CPU usage of segment hosts exceeds a specified percentage for a specified number of minutes
  • Spill files for an active query exceed a specified number of gigabytes
  • Runtime for a query exceeds a specified number of minutes
  • A query is blocked for more than a specified number of minutes

Query Text Download for Long Queries

On the Query Details page, if the query text is longer than 100K characters, Command Center only shows the first 100K characters and, when you click COPY, copies only the first 100K characters to the clipboard. When the query text is longer than 100K characters, Command Center adds a button, Retrieve full query text, which you can click to download a text file containing the entire text of the query. The file is available to download for 24 hours, or until the query is saved to history, when history collection is enabled.

Fixed Issues

  • 161436520 - Panels displaying help text can now be scrolled vertically when the text does not fit in the window.

  • 158278655 - The displayed metrics for a query could be inaccurate if Command Center received the query “done” status before the final metrics arrived. Now the final metrics are updated even when they arrive after the “done” status.

  • 161074984 - Repeated assertion errors logged in the agent.log file caused log files to grow quickly and consume too much disk space. This is fixed. Each type of assertion error will be logged no more than once every 10 minutes.

  • 162114428 - When a user visits Command Center with a browser, the Command Center web server (gpccws process) establishes a websocket connection and runs two goroutines to service the websocket. When the websocket closes, one of the goroutines does not exit. After many connections the gpccws process can occupy gigabytes of memory. This memory leak is fixed.

Enhancements and Changes in Greenplum Command Center 4.4.2

  • The Greenplum Database metrics collector extension is now enabled only when Command Center agents are running. Previously, if an agent process terminated, the metrics collector continued to collect and send data.

Greenplum Command Center 4.4.2 contains the following resolved issues:

  • 160195564 - When Command Center is restarted after a user has submitted a query, and the user submits another query in the same session, the database name and user name are missing from the second query in the Command Center interface. This is fixed.

  • 161077294 - The Authentication view now prevents the user from saving an undefined entry to the pg_hba.conf file.

  • 161274227 - The Command Center installer, gpccinstall, no longer prints errors to the pg_log log file about missing iterators and emc_connect_history tables.

  • 161273865 - The Query Monitor no longer shows runtimes flashing to 0 seconds when a query is cancelled through the user interface.

Enhancements and Changes in Greenplum Command Center 4.4.1

  • The Admin> Authorization view allows entering host names in the Address field.

  • On the visual query plan, a data motion showed a finished status when the send data operation completed. This is changed so that the node completes only after the corresponding data receive operation has also completed.

  • Enabled cross-site request forgery prevention during login.

Resolved Issues in Greenplum Command Center 4.4.1

  • 160679694 - If Command Center is restarted while on the Query Monitor view and the browser is then refreshed, an extra web socket connection is created. This is fixed.

  • MPP-29539 - Command Center agent (ccagent) logging is disabled for a known issue with PL/pgSQL queries.

Enhancements and Changes in Greenplum Command Center 4.4.0

Greenplum Command Center 4.4.0 contains the following features and enhancements.

Workload Management

  • Command Center has a new user interface to assist administrators in enabling resource groups in Greenplum Database, importing existing resource queues to resource groups, and enabling workload management with Command Center. The option to import resource queues to resource groups is presented if no resource groups have been created (other than default_group and admin_group) and Greenplum Database has resource queues to convert (other than pg_default). Once the administrator has imported resource queues, or chosen to skip importing resource queues, the option to import queues is no longer presented.

  • The resource group list on the Workload Management view has a new column to show the minimum (fixed) amount of memory that will be allocated to a query for each resource group. This value is recalculated when you enter new values while editing resource groups.

  • Administrators can now define resource group assignment rules and idle session kill rules with an interactive interface. It is no longer necessary to edit the JSON document for workload management rules. The JSON text field is removed.

  • The Workload Management view changed to a light theme.


  • Command Center users with the Self Only permission level can:

    • see all queries on the query monitor, including queries owned by other users
    • cancel their own queries
    • access the query details view for their own queries
    • hover on a query to see query text for their own queries
    • hover on locking/blocking queries and access details of locking/blocking queries that do not belong to other users

    Users with Self Only permission level cannot see query text or access the details views of others’ queries.

  • The Greenplum Database roles gpcc_basic, gpcc_operator, and gpcc_operator_basic are created during Command Center installation if they do not already exist.

Query Monitor

  • Fixed a bug where the database name and role name were missing from query details when queries are executed in a session after restarting Command Center.

Enhancements and Changes in Greenplum Command Center 4.3.0

Greenplum Command Center 4.3.0 contains the following enhancements.

Resource Group Management

The Admin> Workload Mgmt view has a new user interface you can use to add and remove resource groups and to change the Concurrency, CPU %, and Memory % attributes of resource groups.

Resource Group Role Assignments

The Admin> Workload Mgmt view has a new user interface you can use to view and change Greenplum Database roles’ default resource groups.

Details Added to Visual Query Plan Steps

The metrics collector extension in Greenplum Database release 5.9 is updated to submit additional information about each step in the query plan to the Command Center backend. Command Center displays this information in the visual query plan when you expand a step in the query plan. The new information displayed depends on the operation the step performs and includes details such as hash key, merge key, join condition, or filter condition. Previously, you could only see this information by generating the textual query plan.

Enhancements and Changes in Greenplum Command Center 4.2.0

Greenplum Command Center 4.2.0 contains the following enhancements.

Visual Query Plan

The Command Center Query Details view now includes a visual query plan.

Idle Session Kill Rules

Idle session kill rules can include optional exemptedRoles and message parameters.

  • The value of the exemptedRoles parameter is a list of Greenplum Database roles that are exempted from the rule. The list can include Posix regular expressions to match Greenplum Database role names.
  • The value of the message parameter is a string to include in the message that is displayed when a session is killed.

Command Center Can Run on the Master Host or Standby Master Host

The Greenplum Command Center web server and backend may now be executed on the master host or on the standby master host. Running GPCC on the standby master host is recommended to avoid adding load to the master server, but it is no longer a requirement. After the GPCC software is installed, log in to the host where you want to run GPCC, source the gpcc_path.sh file in the GPCC installation directory, and run the gpcc start command.

Enhancements and Changes in Greenplum Command Center 4.1.0

Greenplum Command Center 4.1.0 contains the following enhancements.

  • Command Center administrators can set permission levels for Command Center users. Permissions are enforced as described in the documentation.
  • On the Query Detail view, clicking Copy in the query text or query plan panel copies the text in the panel to the clipboard.

  • A help icon and in-app help have been added on the Query Monitor and Query Detail views.

The following workload management features, improvements, and bug fixes have been added in the workload management extension included with Greenplum Database 5.8.0.

  • On the Admin> Workload Mgmt view, you can add idle session kill rules for each resource group. When you add these rules, the Greenplum Database workload management extension kills a session after it has been idle for the number of seconds you specify. See Workload Management for syntax and examples.

  • Optimizations have been implemented to reduce the impact on Greenplum Database when the workload management extension is disabled.

  • The workload management extension takes advantage of resource group name-to-id caching added in Greenplum Database.

  • Fixed a bug in the workload management extension that caused errors to print to the psql prompt, even when the extension was disabled.

Greenplum Command Center 4.1.0 contains the following bug fixes.

  • In the History Detail view, when either of the Disk R or Disk W metrics is 0, both are reported to be 0. This is fixed.

  • When using the Kerberos gpmon-only authentication mode, generating an explain plan failed. This is fixed.

  • The Command Center agent failed with the message “Error: can’t find gpcc.query_metrics_port, metrics_collector is not correctly installed.” This occurs when running Command Center on a Greenplum Database system that was upgraded from an earlier Greenplum Database 5.x release. The metrics_collector and gp_wlm extensions are installed with the upgrade, but the upgrade process does not perform the required configuration changes in the postgresql.conf configuration file. The Command Center installation instructions now include steps to manually configure and restart an upgraded Greenplum Database system.

  • When Command Center starts, an error message is written in the Greenplum Database log file: “function gpcc_schema.read_pghba(unknown) does not exist.” This is fixed.

  • A change to time zone handling in Greenplum Database 5.7 can cause Greenplum Command Center 4.0.0 to display an incorrect time if the master host operating system time and Greenplum Database use different time zones. Now Command Center times are displayed using the time zone of the Greenplum Database master host operating system. The current time, last sync time, and timestamps in alert logs, cluster metrics, and query history are all displayed using the master host’s system time zone.

  • On the Admin> Authorization view, when the authorization method is gss and the options field contains text, changing the method to trust does not clear the options field and it is not possible to save changes to the pg_hba.conf file.

  • The Command Center web server, gpccws, spawns ssh processes but does not reap them in a timely manner, leading to many zombie processes. This is fixed.

  • On the Admin> Authentication view, if the number of users listed in the user column is longer than can be displayed, Command Center truncates the list and adds ellipsis (...) to the end. Only administrators can view the complete value, by editing the field. Now any user with access to the view can see the full list of users by hovering over the field.

Enhancements and Changes in Greenplum Command Center 4.0.0

Command Center Installation Changes

Greenplum Command Center 4.x software, unlike previous releases, is installed on every host in the Greenplum Database cluster. The Command Center web server and backend run on the standby master, if your Greenplum Database cluster has a standby master. If there is no standby master, Command Center runs on the master host instead.

To modify the Command Center installation—for example to enable or disable SSL or install Command Center on new or replaced hosts—just re-execute the installer and restart Command Center. It is not necessary to uninstall Command Center before reinstalling.

There is one Command Center installation per Greenplum Database cluster. It is no longer necessary to create Command Center instances after installing the software.

The gpcmdr command-line utility is replaced with the new gpcc utility. Use the gpcc utility to start and stop Command Center and metrics collection agents, check Command Center status, and enable or disable Kerberos authentication.

In previous releases, the gpmon role required only local connections to databases on the Greenplum master host. In Greenplum Command Center 4.x, the gpmon user must be able to connect to databases from the host running the Command Center web server and backend, which requires adding a host entry to the pg_hba.conf authentication configuration file.

Real-time Query Metrics

Greenplum Command Center 4.0 introduces real-time query metrics for Pivotal Greenplum Database 5.7 and above. This new feature combines the following new features in Greenplum Database and Greenplum Command Center:

  • Greenplum Database saves query execution metrics in shared memory while queries execute.
  • A new Greenplum Database metrics collection extension, included with Pivotal Greenplum Database, emits the saved metrics as UDP datagrams.
  • A new Greenplum Command Center metrics collection agent running on each Greenplum Database host receives the datagrams and posts metrics to the Greenplum Command Center backend. The Command Center backend starts and manages the metrics collection agents.

The Command Center Query monitor view updates in real time so you can see queries that are waiting to execute and the current status and resource usage for queries that are running.

Metrics collection now includes lock and spill file information. On the Query Monitor, you can see which queries are blocked and which queries hold the locks blocking them. The Query Monitor shows the total size of spill files created on all segments for each query.

Installing the gpperfmon database remains a prerequisite for Command Center. The gpperfmon database is the source for query history displayed in the Command Center user interface. The new real-time metrics are not persisted and are not directly related to the metrics collected and persisted in the gpperfmon database.

Workload Management

Workload management is now an integrated Command Center feature rather than a separate product. Workload management is available in Command Center only after resource groups have been enabled in Greenplum Database by changing the gp_resource_manager server configuration parameter from 'queue' to 'group' and enabling Linux control groups (cgroups).

In Command Center 4.0, workload management allows you to assign transactions to Greenplum Database resource groups at execution time by evaluating the current database role and query tags against workload assignment filters you define in Command Center. Query tags are user-defined name=value parameters that you define in the gpcc.query_tags database session parameter. You can define multiple query tags separated by semicolons. Set query tags in a Greenplum Database session either as a parameter in the database connection URL or by executing SET gpcc.query_tags TO '<tag1>=<val1>;<tag2>=<val2>;...' in the database session.

When a transaction is about to execute, the current database role and query tags are compared to the workload assignment filters that you have created in Command Center. If a match is found, the transaction is assigned to a resource group according to the workload management filter. Otherwise, the transaction is assigned to the database user’s resource group, which is the default behavior when Command Center workload management is not enabled.

Workload management uses the gp_wlm database extension included with Pivotal Greenplum Database.

Unimplemented Features

Some features available in previous Greenplum Command Center releases have been removed or are not yet implemented in Command Center 4.x.

  • The ability for a Command Center admin to post a message to the Query Monitor view is not yet implemented.
  • The multi-cluster view has been removed.

Known Issues

The following are known issues in the current Greenplum Command Center release.

Failure to Auto-Create Monthly Partitions in gpmetrics Schema

In time zones with daylight savings time (DST) ending in the month of October, Greenplum Command Center fails to create new October partitions for tables in the gpmetrics schema because it specifies the same start and end date for the partitions. The error does not occur in time zones that do not have DST, or in time zones with DST ending in November.


Manually create partitions for October 2019.

Log in to the Greenplum master server as the gpadmin user and connect to the gpperfmon database as the gpmon user:

psql -U gpmon gpperfmon

Enter the following commands to create the October 2019 partitions:

ALTER TABLE gpmetrics.gpcc_disk_history
      ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE

ALTER TABLE gpmetrics.gpcc_system_history
      ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE

ALTER TABLE gpmetrics.gpcc_queries_history
      ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE

ALTER TABLE gpmetrics.gpcc_plannode_history
      ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE

ALTER TABLE gpmetrics.gpcc_database_history
      ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE

ALTER TABLE gpmetrics.gpcc_alert_log
      ADD PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE

ALTER TABLE gpmetrics.gpcc_pg_log_history
      SPLIT DEFAULT PARTITION START (date '2019-10-01') INCLUSIVE END (date '2019-11-01') EXCLUSIVE

A Large Memory Leak Occurs Over Time When Certain Alerts are Enabled

[MPP-29942] A deadlock can occur when reading incoming metrics if any of the following alerts are enabled:

  • Average memory (segment hosts) exceeds % for min
  • Memory (master) exceeds % for min
  • Average CPU (segment hosts) exceeds % for min
  • CPU (master) exceeds % for min

The deadlock causes incoming metrics to be blocked and accumulated in memory.

To work around this issue, disable all of the above alerts and restart Command Center.

This issue exists in Greenplum Command Center 4.6.1 and 4.7.0.

Unable to View Real Time Queries After Upgrading From a Previous Command Center Release

If you install a new version of Greenplum Command Center using the same port number as the previous version, and you use the Chrome web browser, you may be unable to view real-time queries until after you clear the browser’s cache. See the note in Connecting to the Command Center Console for steps to clear the browser cache.

Command Center User Interface is Slow on Internet Explorer 11

When running Greenplum Command Center in the IE 11 browser on Windows, the user interface can be very slow when the History page search returns a large number of rows. This is a known issue with the React JavaScript library and IE 11. Pivotal recommends using the Chrome browser to run Command Center on Windows.