Configuration

The Configuration page describes SQream’s method for configuring your instance of SQream and includes the following topics:

Overview

Modifications that you make to your configurations are persistent based on whether they are made at the session or cluster level. Persistent configurations are modifications made to attributes that are retained after shutting down your system.

Configuring Your Instance of SQream

The Configuring Your Instance of SQream section describes the following:

Configuring SQream Using the SQream Configuration File

You can configure your instance of SQream using the SQream configuration file (config.json). This file displays all the parameters required for configuring your instance of SQream and are read-only. The parameter settings in this file are stored at the metadata level and are applied globally to all workers connected to it.

The following is an example of a worker configuration file:

{
    “cluster”: “/home/test_user/sqream_testing_temp/sqreamdb”,
    “gpu”:  0,
    “licensePath”: “home/test_user/SQream/tests/license.enc”,
    “machineIP”: “127.0.0.1”,
    “metadataServerIp”: “127.0.0.1”,
    “metadataServerPort”: “3105,
    “port”: 5000,
    “useConfigIP”” true,
    “legacyConfigFilePath”: “home/SQream_develop/SqrmRT/utils/json/legacy_congif.json”
}

You can access the SQream configuration file from the legacyConfigFilePath parameter shown above.

Configuring SQream Using a Legacy Configuration File

The legacy configuration file provides access to the read/write flags used in SQream’s previous configuration method. A link to this file is provided in the legacyConfigFilePath parameter in the worker configuration file.

The following is an example of the legacy configuration file:

{
   “developerMode”: true,
   “reextentUse”: false,
   “useClientLog”: true,
   “useMetadataServer”” false
}

Session-Based Configuration

Session-based configurations are not persistent and are deleted when your session ends. This method enables you to modify all required configurations while avoiding conflicts between flag attributes modified on different devices at different points in time.

The SET flag_name command is used to modify flag attributes. Any modifications you make with the SET flag_name command apply only to the open session, and are not saved when the session ends.

For more information, see the following:

Cluster-Based Configuration

SQream uses cluster-based configuration, enabling you to centralize configurations for all workers on the cluster. Only flags set to the regular or cluster flag type have access to cluster-based configuration. Configurations made on the cluster level are persistent.

For more information, see the following:

For more information on flag-based access to cluster-based configuration, see Configuration Flag Types below.

Configuration Flag Types

The flag type attribute can be set for each flag and determines its write access as follows:

  • Regular: session-based read/write flags that can be stored in the metadata file
  • Cluster: global cluster-based read/write flags that can be stored in the metadata file
  • Worker: single worker-based read-only flags that can be stored in the worker configuration file

The flag type determines which files can be accessed and which commands or commands sets users can run.

The following table describes the file or command modification rights for each flag type:

Flag Type Legacy Configuration File ALTER SYSTEM SET Worker Config File
Regular Can modify Can modify Cannot modify
Cluster Cannot modify Can modify Cannot modify
Worker Cannot modify Cannot modify Can modify

Configuration Commands

The configuration commands are associated with particular flag types based on permissions.

The following table describes the commands or command sets that can be run based on their flag type:

Flag Type Command Description Example
Regular SET <flag_name> Used for modifying flag attributes. SET developerMode=true
Cluster ALTER SYSTEM SET <flag-name> Used to storing or modifying flag attributes in the metadata file. ALTER SYSTEM SET <heartbeatInterval=12;>
Cluster ALTER SYSTEM RESET <flag-name | ALL> Used to remove a flag or all flag attributes from the metadata file. ALTER SYSTEM RESET <heartbeatInterval ALTER SYSTEM RESET ALL>
Regular, Cluster, Worker SHOW <flag-name> | ALL Used to print the value of a specified value or all flag values. SHOW <heartbeatInterval>
Regular, Cluster, Worker SHOW ALL LIKE Used as a wildcard character for flag names. SHOW <heartbeat*>
Regular, Cluster, Worker show_conf UF

Used to print all flags with the following attributes:

  • Flag name
  • Default value
  • Is developer mode (Boolean)
  • Flag category
  • Flag type
rechunkThreshold,90,true,RND,regular
Regular, Cluster, Worker show_conf_extended UF Used to print all information output by the show_conf UF command, in addition to description, usage, data type, default value and range. spoolMemoryGB,15,false,generic,regular,Amount of memory (GB) the server can use for spooling,”Statement that perform “”group by””, “”order by”” or “”join”” operation(s) on large set of data will run much faster if given enough spool memory, otherwise disk spooling will be used resulting in performance hit.”,uint,,0-5000
Regular, Cluster, Worker show_md_flag UF Used to show a specific flag/all flags stored in the metadata file.
  • Example 1: * master=> ALTER SYSTEM SET heartbeatTimeout=111;
  • Example 2: * master=> select show_md_flag(‘all’); heartbeatTimeout,111
  • Example 3: * master=> select show_md_flag(‘heartbeatTimeout’); heartbeatTimeout,111

Configuration Roles

SQream divides flags into the following roles, each with their own set of permissions:

  • Generic – Flags that can be modified by standard users on a session basis.
  • Admin – Flags that can be modified by administrators on a session and cluster basis using the ALTER SYSTEM SET command.

The following table describes the Generic and Admin configuration flags:

Showing All Flags in the Catalog Table

SQream uses the sqream_catalog.parameters catalog table for showing all flags, providing the scope (default, cluster and session), description, default value and actual value.

The following is the correct syntax for a catalog table query:

SELECT * FROM sqream_catalog.settings

The following is an example of a catalog table query:

externalTableBlobEstimate, 100, 100, default,
varcharEncoding, ascii, ascii, default, Changes the expected encoding for Varchar columns
useCrcForTextJoinKeys, true, true, default,
hiveStyleImplicitStringCasts, false, false, default,