sqream-console

sqream-console is an interactive shell designed to help manage a dockerized SQream DB installation.

The console itself is a dockerized application.

This page serves as a reference for the options and parameters.

Starting the console 

sqream-console can be found in your SQream DB installation, under the name sqream-console.

Start the console by executing it from the shell

$ ./sqream-console
....................................................................................................................

███████╗ ██████╗ ██████╗ ███████╗ █████╗ ███╗   ███╗     ██████╗ ██████╗ ███╗   ██╗███████╗ ██████╗ ██╗     ███████╗
██╔════╝██╔═══██╗██╔══██╗██╔════╝██╔══██╗████╗ ████║    ██╔════╝██╔═══██╗████╗  ██║██╔════╝██╔═══██╗██║     ██╔════╝
███████╗██║   ██║██████╔╝█████╗  ███████║██╔████╔██║    ██║     ██║   ██║██╔██╗ ██║███████╗██║   ██║██║     █████╗
╚════██║██║▄▄ ██║██╔══██╗██╔══╝  ██╔══██║██║╚██╔╝██║    ██║     ██║   ██║██║╚██╗██║╚════██║██║   ██║██║     ██╔══╝
███████║╚██████╔╝██║  ██║███████╗██║  ██║██║ ╚═╝ ██║    ╚██████╗╚██████╔╝██║ ╚████║███████║╚██████╔╝███████╗███████╗
╚══════╝ ╚══▀▀═╝ ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝     ╚═════╝ ╚═════╝ ╚═╝  ╚═══╝╚══════╝ ╚═════╝ ╚══════╝╚══════╝

....................................................................................................................


Welcome to SQream Console ver 1.7.6, type exit to log-out

usage: sqream [-h] [--settings] {master,worker,client,editor} ...

Run SQream Cluster

optional arguments:
  -h, --help            show this help message and exit
  --settings            sqream environment variables settings

subcommands:
  sqream services

  {master,worker,client,editor}
                        sub-command help
    master              start sqream master
    worker              start sqream worker
    client              operating sqream client
    editor              operating sqream statement editor
sqream-console>

The console is now waiting for commands.

The console is a wrapper around a standard linux shell. It supports commands like ls, cp, etc.

All SQream DB-specific commands start with the keyword sqream.

Operations and flag reference 

Commands 

Command	Description
`sqream --help`	Shows the initial usage information
`sqream master`	Controls the master node’s operations
`sqream worker`	Controls workers’ operations
`sqream client`	Access to sqream sql
`sqream editor`	Controls the statement editor’s operations (web UI)

Master 

The master node contains the metadata server and the load balancer.

Syntax 

sqream master <flags>

Flag/command	Description
`--start [ --single-host ]`	Starts the master node. The `--single-host` modifier sets the mode to allow all containers to run on the same server.
`--stop [ --all ]`	Stops the master node and all connected workers. The `--all` modifier instructs the `--stop` command to stop all running services related to SQream DB
`--list`	Shows a list of all active master nodes and their workers
`-p <port>`	Sets the port for the load balancer. Defaults to `3108`
`-m <port>`	Sets the port for the metadata server. Defaults to `3105`

Common usage 

Start master node 

sqream-console> sqream master --start
starting master server in single_host mode ...
sqream_single_host_master is up and listening on ports:   3105,3108

Start master node on different ports 

sqream-console> sqream master --start -p 4105 -m 4108
starting master server in single_host mode ...
sqream_single_host_master is up and listening on ports:   4105,4108

Listing active master nodes and workers 

sqream-console> sqream master --list
container name: sqream_single_host_worker_1, container id: de9b8aff0a9c
container name: sqream_single_host_worker_0, container id: c919e8fb78c8
container name: sqream_single_host_master, container id: ea7eef80e038

Stopping all SQream DB workers and master 

sqream-console> sqream master --stop --all
  shutting down 2 sqream services ...
 sqream_editor    stopped
 sqream_single_host_worker_1    stopped
 sqream_single_host_worker_0    stopped
 sqream_single_host_master    stopped

Workers 

Workers are SQream DB daemons, that connect to the master node.

Syntax 

sqream worker <flags>

Flag/command	Description
`--start [ options [ ...] ]`	Starts worker nodes. See options table below.
`--stop [ <worker name> \| --all ]`	Stops the specified worker name. The `--all` modifier instructs the `--stop` command to stop all running workers.

Start options are specified consecutively, separated by spaces.

Start options
Option	Description
`<n>`	Specifies the number of workers to start
`-j <config file> [ ...]`	Specifies configuration files to apply to each worker. When launching multiple workers, specify one file per worker, separated by spaces.
`-p <port> [ ...]`	Sets the ports to listen on. When launching multiple workers, specify one port per worker, separated by spaces. Defaults to 5000 - 5000+n.
`-g <gpu id> [ ...]`	Sets the GPU ordinal to assign to each worker. When launching multiple workers, specify one GPU ordinal per worker, separated by spaces. Defaults to automatic allocation.
`-m <spool memory>`	Sets the spool memory per node in gigabytes.
`--master-host`	Sets the hostname for the master node. Defaults to `localhost`.
`--master-port`	Sets the port for the master node. Defaults to `3105`.
`--stand-alone`	For testing only: Starts a worker without connecting to the master node.

Common usage 

Start 2 workers 

After starting the master node, start workers:

sqream-console> sqream worker --start 2
started sqream_single_host_worker_0 on port 5000, allocated gpu: 0
started sqream_single_host_worker_1 on port 5001, allocated gpu: 1

Stop a single worker 

To stop a single worker, find its name first:

sqream-console> sqream master --list
container name: sqream_single_host_worker_1, container id: de9b8aff0a9c
container name: sqream_single_host_worker_0, container id: c919e8fb78c8
container name: sqream_single_host_master, container id: ea7eef80e038

Then, issue a stop command:

sqream-console> sqream worker --stop sqream_single_host_worker_1
stopped sqream_single_host_worker_1

Start workers with a different spool size 

If no spool size is specified, the RAM is equally distributed among workers. Sometimes a system engineer may wish to specify the spool size manually.

This example starts two workers, with a spool size of 50GB per node:

sqream-console> sqream worker --start 2 -m 50

Starting multiple workers on non-dedicated GPUs 

By default, SQream DB workers assign one worker per GPU. However, a system engineer may wish to assign multiple workers per GPU, if the workload permits it.

This example starts 4 workers on 2 GPUs, with 50GB spool each:

sqream-console> sqream worker --start 2 -g 0 -m 50
started sqream_single_host_worker_0 on port 5000, allocated gpu: 0
started sqream_single_host_worker_1 on port 5001, allocated gpu: 0
sqream-console> sqream worker --start 2 -g 1 -m 50
started sqream_single_host_worker_2 on port 5002, allocated gpu: 1
started sqream_single_host_worker_3 on port 5003, allocated gpu: 1

Overriding default configuration files 

It is possible to override default configuration settings by listing a configuration file for every worker.

This example starts 2 workers on the same GPU, with modified configuration files:

sqream-console> sqream worker --start 2 -g 0 -j /etc/sqream/configfile.json /etc/sqream/configfile2.json

Client 

The client operation runs sqream sql in interactive mode.

Note

The dockerized client is useful for testing and experimentation. It is not the recommended method for executing analytic queries. See more about connecting a third party tool to SQream DB for data analysis.

Syntax 

sqream client <flags>

Flag/command	Description
`--master`	Connects to the master node via the load balancer
`--worker`	Connects to a worker directly
`--host <hostname>`	Specifies the hostname to connect to. Defaults to `localhost`.
`--port <port>`, `-p <port>`	Specifies the port to connect to. Defaults to `3108` when used with `-master`.
`--user <username>`, `-u <username>`	Specifies the role’s username to use
`--password <password>`, `-w <password>`	Specifies the password to use for the role
`--database <database>`, `-d <database>`	Specifies the database name for the connection. Defaults to `master`.

Common usage 

Start a client 

Connect to default master database through the load balancer:

sqream-console> sqream client --master -u sqream -w sqream
Interactive client mode
To quit, use ^D or \q.

master=> _

Start a client to a specific worker 

Connect to database raviga directly to a worker on port 5000:

sqream-console> sqream client --worker -u sqream -w sqream -p 5000 -d raviga
Interactive client mode
To quit, use ^D or \q.

raviga=> _

Start master node on different ports 

sqream-console> sqream master --start -p 4105 -m 4108
starting master server in single_host mode ...
sqream_single_host_master is up and listening on ports:   4105,4108

Listing active master nodes and worker nodes 

sqream-console> sqream master --list
container name: sqream_single_host_worker_1, container id: de9b8aff0a9c
container name: sqream_single_host_worker_0, container id: c919e8fb78c8
container name: sqream_single_host_master, container id: ea7eef80e038

Editor 

The editor operation runs the web UI for the SQream DB Statement Editor.

The editor can be used to run queries from a browser.

Syntax 

sqream editor <flags>

Flag/command	Description
`--start`	Start the statement editor
`--stop`	Shut down the statement editor
`--port <port>`, `-p <port>`	Specify a different port for the editor. Defaults to `3000`.

Common usage 

Start the editor UI 

sqream-console> sqream editor --start
access sqream statement editor through Chrome http://192.168.0.100:3000

Stop the editor UI 

sqream-console> sqream editor --stop
 sqream_editor    stopped

Using the console to start SQream DB 

The console is used to start and stop SQream DB components in a dockerized environment.

Starting a SQream DB cluster for the first time 

To start a SQream DB cluster, start the master node, followed by workers.

The example below starts 2 workers, running on 2 dedicated GPUs.

sqream-console> sqream master --start
starting master server in single_host mode ...
sqream_single_host_master is up and listening on ports:   3105,3108

sqream-console> sqream worker --start 2
started sqream_single_host_worker_0 on port 5000, allocated gpu: 0
started sqream_single_host_worker_1 on port 5001, allocated gpu: 1

sqream-console> sqream editor --start
access sqream statement editor through Chrome http://192.168.0.100:3000

SQream DB is now listening on port 3108 for any incoming statements.

A user can also access the web editor (running on port 3000 on the SQream DB machine) to connect and run queries.