Open search page
Hello, I’m DocuDroid!
Open Ask AI page
Submitting feedback
Thank you for rating our AI Search!
We would be grateful if you could share your thoughts so we can improve our AI Search for you and other readers.
GitHub

Configure memory and threading

Anton Monakov

A single PXF service (JVM) serves multiple segments on a segment host. Under concurrent workloads or with queries against large files, the PXF heap size can become a limiting runtime factor, with queries hanging or failing due to insufficient memory, or the Java garbage collector impacting response times.

To address these issues, you can increase the Java maximum heap size or decrease the Tomcat maximum number of threads. You can also configure PXF to auto-terminate the server (activated by default) or dump the JVM heap when it detects an out-of-memory (OOM) condition.

Increase the JVM heap size

Each PXF service running on a Greengage DB host is configured with a default maximum JVM heap size of 2 GB and an initial heap size of 1 GB. If the amount of RAM on the Greengage DB cluster hosts allows it, try increasing the maximum heap size. Under heavy workload, it is recommended to set the initial and maximum heap size to the same value.

To increase the heap size for the PXF service running on each host in a Greengage DB cluster:

  1. On the Greengage DB master host, log in as gpadmin.

  2. Open the $PXF_BASE/conf/pxf-env.sh file in a text editor and update the -Xmx and -Xms options of the PXF_JVM_OPTS setting to the desired value, for example:

    PXF_JVM_OPTS="-Xmx3g -Xms3g"

  3. Save and close the file.

  4. Synchronize the PXF configuration to the Greengage DB cluster and then restart PXF on all hosts:

    $ pxf cluster sync
$ pxf cluster restart

Configure out-of-memory condition actions

If an out-of-memory condition occurs, PXF returns the following error in response to a query:

ERROR:  PXF server error : JVM heap space

You can configure the PXF JVM to perform the following actions in case of an OOM:

Auto-terminate the PXF server

By default, PXF is configured to auto-terminate the PXF service on a Greengage DB host when it detects an OOM. The auto-terminate behavior is regulated by the PXF_OOM_KILL environment variable in the $PXF_BASE/conf/pxf-env.sh configuration file.

When auto-terminate is activated and the PXF JVM terminates the PXF service on the host in an out-of-memory situation, the following actions are performed:

  1. PXF logs the following messages to $PXF_LOGDIR/pxf-oom.log on the segment host:

    =====> <date> PXF Out of memory detected <======
=====> <date> PXF shutdown scheduled <======
=====> <date> Stopping PXF <======

  2. Any query run on a PXF external table fails with the following error until the PXF service is restarted on the host:

    ... Failed to connect to <host> port 5888: Connection refused

When the PXF service on a host is shut down due to an OOM condition, you must explicitly restart it by running the pxf start command. For more information on the available PXF commands, see Overview of the pxf commands.

Dump the JVM heap

In an out of memory situation, it may be useful to capture the JVM heap dump to help determine the failure cause. You can configure PXF to write the heap dump to a file by setting the PXF_OOM_DUMP_PATH environment variable in the $PXF_BASE/conf/pxf-env.sh configuration file.

By default, heap dumping on OOM is not activated. To activate it, set PXF_OOM_DUMP_PATH to the absolute path to a file or directory. Ensure that the gpadmin user has write access to the dump file or directory.

  • If you specify a directory, the heap dump is written to the <directory>/java_pid<pid>.hprof file, where <pid> identifies the process ID of the PXF service instance. The PXF JVM writes a new file to the directory on every new OOM condition.

  • If you specify a file that does not exist, the PXF JVM creates the file and writes the heap dump to it. If the file already exists, the JVM will not dump the heap.

NOTE

Heap dump files are often rather large. If you activate heap dumping on OOM and specify a directory for PXF_OOM_DUMP_PATH, multiple OOMs will generate multiple files in the directory and could potentially consume a large amount of disk space. If you specify a file for PXF_OOM_DUMP_PATH, disk usage is constant when the file name does not change. You must rename the dump file or configure a different PXF_OOM_DUMP_PATH to generate subsequent heap dumps.

Configure default actions for an OOM condition

Auto-termination of the PXF service on OOM is activated by default. Heap dump generation on OOM is deactivated by default.

To configure these actions:

  1. On the Greengage DB master host, log in as gpadmin.

  2. Open the $PXF_BASE/conf/pxf-env.sh file in a text editor.

    • To configure auto-termination of the PXF service on OOM, uncomment the PXF_OOM_KILL property and update the value. For example, to turn auto-termination off, set the value to false:

      export PXF_OOM_KILL=false

    • To configure automatic heap dumping on OOM, uncomment the PXF_OOM_DUMP_PATH property and update the value. To turn the action on, set the PXF_OOM_DUMP_PATH property value to the file system location where the PXF JVM will dump the JVM heap. For example, to dump to a file named /home/gpadmin/pxf_oom_dump:

      export PXF_OOM_DUMP_PATH=/home/gpadmin/pxf_oom_dump

      To turn off heap dumping, comment out the PXF_OOM_DUMP_PATH property setting:

      #export PXF_OOM_DUMP_PATH=/home/gpadmin/pxf_oom_dump

  3. Save and close the file.

  4. Synchronize the PXF configuration to the Greengage DB cluster and then restart PXF on all hosts:

    $ pxf cluster sync
$ pxf cluster restart

Adjust the number of Tomcat working threads

If increasing the maximum heap size is not suitable for your Greengage DB cluster, you can adjust the number of concurrent working threads for the PXF embedded Tomcat web server. A decrease in the number of running threads will prevent any PXF server from exhausting its memory while ensuring that current queries run to completion (though slower). Tomcat’s default behavior is to queue requests until a thread is free or the queue is empty.

The default maximum number of Tomcat threads for PXF is 200. This value is controlled by the pxf.max.threads property in the pxf-application.properties configuration file. An increase in the thread count correlates with an increase in memory consumption. When working with a large number of files in an external Hive data store or reading compressed ORC or Parquet data, consider specifying a lower pxf.max.threads value, which will limit concurrency and memory consumption.

To set the maximum number of Tomcat threads for the PXF service running on each host in a Greengage DB cluster:

  1. On the Greengage DB master host, log in as gpadmin.

  2. Open the $PXF_BASE/conf/pxf-application.properties file in a text editor, uncomment the pxf.max.threads property and update it to the desired value. For example, to reduce the maximum number of Tomcat threads to 100:

    pxf.max.threads=100

  3. Save and close the file.

  4. Synchronize the PXF configuration to the Greengage DB cluster and then restart PXF on all hosts:

    $ pxf cluster sync
$ pxf cluster restart
Previous
JDBC
Next
Logging