Configure logging in PXF

Anton Monakov

Contents

Configure the log directory
Configure service-level logging
- Configure the log level for a specific host
- Configure the log level for an entire cluster
Configure client-level logging

PXF supports two categories of message logging: service-level and client-level.

Service-level logging is managed by PXF and provides the following log levels (from most to least severe):

FATAL
ERROR
WARN
INFO
DEBUG
TRACE

By default, the PXF service logs messages of the INFO and more severe levels. For some third-party libraries, only the messages of the WARN and more severe levels are logged to reduce verbosity.

PXF captures messages written to stdout and stderr and writes them to the $PXF_LOGDIR/pxf-app.out file. The file may contain service startup messages captured before logging is fully configured as well as debug output messages. By default, the messages logged after startup are written to the $PXF_LOGDIR/pxf-service.log file. If needed, you can change the PXF log directory as described in Configure the log directory.

Client-level logging is managed by a specific Greengage DB client application. See Configure client-level logging for details on configuring client-level logging for the psql utility.

By default, PXF logs messages to the $PXF_LOGDIR directory, with the PXF_LOGDIR environment variable defined as $PXF_BASE/logs.

To change the PXF log directory:

On the Greengage DB master host, log in as gpadmin.
Open the $PXF_BASE/conf/pxf-env.sh file in a text editor and set the desired log directory path as the PXF_LOGDIR variable value. The provided log directory must exist on all Greengage DB hosts and be accessible by the gpadmin user.
```
export PXF_LOGDIR="/new/log/dir"
```
Synchronize the PXF configuration to the Greengage DB cluster and then restart PXF on all hosts:
```
$ pxf cluster sync
$ pxf cluster restart
```

PXF utilizes Apache Log4j 2 for service-level logging. The PXF service-related log messages are stored in the $PXF_LOGDIR/pxf-app.out and the $PXF_LOGDIR/pxf-service.log files.

By default, the PXF service logs the messages with the INFO and more severe levels. You can change the logging level on a single Greengage DB host or on all hosts in the Greengage DB cluster.

NOTE

While the logs provided by the DEBUG and TRACE levels are more detailed, they are also quite verbose and affect both performance and storage space. After the required information is collected, it is recommended to lower the logging level.

On the Greengage DB master host, log in as gpadmin.
Do one of the following:
- Set the log level in the pxf restart command from the command line. For example, to change the log level from INFO to DEBUG, run the following command:
  $ PXF_LOG_LEVEL=debug pxf restart
- Set the log level in the $PXF_BASE/conf/pxf-application.properties file:
  1. Open the pxf-application.properties file in a text editor and set the desired log level in the pxf.log.level property. For example, to change the log level from INFO to DEBUG, set DEBUG as the pxf.log.level property value:
    
    pxf.log.level=DEBUG
  2. Restart PXF on the host:
    
    $ pxf restart
After running some operations with the PXF service, examine the log messages in the pxf-service.log file.

To revert the log level to INFO, do one of the following:

If the log level has been set via the command line, restart PXF on the host:
```
$ pxf restart
```
If the log level has been set in the pxf-application.properties file, comment out the line or set the pxf.log.level property value back to INFO and then restart PXF on the host:
```
$ pxf restart
```

To change the log level for the PXF service running on every host in the Greengage DB cluster:

On the Greengage DB master host, log in as gpadmin.
Open the $PXF_BASE/conf/pxf-application.properties file in a text editor and set the desired log level as the pxf.log.level property value:
```
pxf.log.level=DEBUG
```
Synchronize the PXF configuration to the Greengage DB cluster and then restart PXF on all hosts:
```
$ pxf cluster sync
$ pxf cluster restart
```
After running some operations with the PXF service, examine the log messages in the pxf-service.log file.

To revert the log level to INFO, comment out the line or set the pxf.log.level property value back to INFO, synchronize the PXF configuration to the Greengage DB cluster, and then restart PXF on all hosts.

Database-level client session logging may aid with troubleshooting the internal PXF service operations. To enable Greengage DB client (psql) debug message logging, set the client_min_messages server configuration parameter to the required logging level.

The following levels are available (from most to least severe):

ERROR
WARNING
NOTICE
LOG
DEBUG1
DEBUG2
DEBUG3
DEBUG4
DEBUG5

The provided logging configuration writes messages to stdout and applies to all operations performed during the session, including operations on PXF external tables.

For example:

SET client_min_messages=DEBUG2;
SELECT * FROM customers;

The output should look similar to the following:

DEBUG1:  Message type Q received by from libpq, len = 31
LOG:  statement: SELECT * FROM customers;
DEBUG2:
[OPT]: Using default search strategy
DEBUG1:  GPORCA produced plan
DEBUG1:  Query plan size to dispatch: 1KB
DEBUG2:  add icid 8 cid 28 status 1
DEBUG1:  Message type M received by from libpq, len = 739  (seg0 slice1 10.92.51.161:10500 pid=1567361)
DEBUG1:  Message type M received by from libpq, len = 739  (seg1 slice1 10.92.51.161:10501 pid=1567360)
...
DEBUG1:  TeardownUDPIFCInterconnect successful
DEBUG1:  Interconnect State: isSender 1 isReceiver 0 snd_queue_depth 2 recv_queue_depth 4
Gp_max_packet_size 8192 UNACK_QUEUE_RING_SLOTS_NUM 2000 TIMER_SPAN 5000 DEFAULT_RTT 20000
hasErrors 0, ic_instance_id 8 ic_id_last_teardown 8 snd_buffer_pool.count 0
snd_buffer_pool.maxCount 0 snd_sock_bufsize 2097152 recv_sock_bufsize 2097152 snd_pkt_count 1
retransmits 0 crc_errors 0 recv_pkt_count 0 recv_ack_num 1 recv_queue_size_avg -nan capacity_avg -nan
freebuf_avg 2.000000 mismatch_pkt_num 0 disordered_pkt_num 0
duplicated_pkt_num 0 rtt/dev [17541/4303, 17541.000000/4303.000000, 17541/4303]  cwnd 2.000000
status_query_msg_num 0  (seg3 slice1 10.92.51.96:10501 pid=1566326)
DEBUG1:  TeardownUDPIFCInterconnect successful  (seg3 slice1 10.92.51.96:10501 pid=1566326)
 id | first_name | last_name |         email          |      address
----+------------+-----------+------------------------+-------------------
  1 | John       | Doe       | john.doe@example.com   | 123 Elm Street
  2 | Jane       | Smith     | jane.smith@example.com | 456 Oak Street
  3 | Bob        | Brown     | bob.brown@example.com  | 789 Pine Street
  4 | Rob        | Stuart    | rob.stuart@example.com | 119 Willow Street
(4 rows)

NOTE

The DEBUG2 and more verbose client session logging levels impact performance. After the required information is collected, it is recommended to lower the logging level, for example:

SET client_min_messages=NOTICE;

Memory and threading

pxf

Configure logging in PXF

Configure the log directory

Configure service-level logging

Configure the log level for a specific host

Configure the log level for an entire cluster

Configure client-level logging