Start a local SQream DB cluster with Docker

See Release Notes to learn about what’s new in the latest release of SQream DB. To upgrade to this release, see Upgrading SQream DB with Docker.

SQream DB is installed on your hosts with NVIDIA Docker. There are several preparation steps to ensure before installing SQream DB, so follow these instructions carefully.

Note

Installing SQream DB requires a license key. Go to SQream Support or contact your SQream account manager for your license key.

Preparing your machine for NVIDIA Docker

To install NVIDIA Docker, we must first install the NVIDIA driver.

Note

SQream DB works best on NVIDIA Tesla series GPUs, which provide better reliability, performance, and stability. The instructions below are written for NVIDIA Tesla GPUs, but other NVIDIA GPUs may work.

CentOS 7 / RHEL 7 / Amazon Linux

Recommended

Follow the installation instructions on NVIDIA’s CUDA Installation Guide for full instructions suitable for your platform. The information listed below is a summary of the necessary steps, and does not cover the full range of options available.

1. Enable EPEL

   EPEL provides additional open-source and free packages from the RHEL ecosystem. The NVIDIA driver depends on packages such as DKMS and libvdpau which are only available on third-party repositories, such as EPEL.

   $ sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
   

1. Install the kernel headers and development tools necessary to compile the NVIDIA driver

   $ sudo yum -y install kernel-devel-$(uname -r) kernel-headers-$(uname -r) gcc
   

2. Install the CUDA repository and install the latest display driver

   $ sudo rpm -Uvh https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-10.1.243-1.x86_64.rpm
   $ sudo yum install -y nvidia-driver-latest
   

Note

If Linux is running with X, switch to text-only mode before installing the display driver.

$ sudo systemctl set-default multi-user.target

This change permanently disables X. If you need X, change set-default to isolate. This will re-enable X on the next reboot.

1. Restart your machine

   ``sudo reboot``
   

2. Verify the installation completed correctly, by asking nvidia-smi, NVIDIA’s system management interface application, to list the available GPUs.

   $ nvidia-smi -L
   GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-...)
   GPU 1: Tesla V100-PCIE-16GB (UUID: GPU-...)
   

3. Enable NVIDIA’s persistence daemon. This is mandatory for IBM POWER, but is recommended for other platforms as well.

   $ sudo systemctl enable nvidia-persistenced && sudo systemctl start nvidia-persistenced
   

   Important

   On POWER9 systems only, disable the udev rule for hot-pluggable memory probing.

   For Red Hat 7 this rule can be found in /lib/udev/rules.d/40-redhat.rules

   For Ubuntu, this rule can be found in in /lib/udev/rules.d/40-vm-hotadd.rules The rule generally takes a form where it detects the addition of a memory block and changes the ‘state’ attribute to online. For example, in RHEL7, the rule looks like this:

   SUBSYSTEM=="memory", ACTION=="add", PROGRAM="/bin/uname -p", RESULT!="s390*", ATTR{state}=="offline", ATTR{state}="online"

   This rule must be disabled by copying the file to /etc/udev/rules.d and commenting out, removing, or changing the hot-pluggable memory rule in the /etc copy so that it does not apply to NVIDIA devices on POWER9.

   • On RHEL 7.5 or earlier versions:

     $ sudo cp /lib/udev/rules.d/40-redhat.rules /etc/udev/rules.d
     $ sudo sed -i '/SUBSYSTEM=="memory", ACTION=="add"/d' /etc/udev/rules.d/40-redhat.rules
     

   • On RHEL 7.6 and later versions:

     $ sudo cp /lib/udev/rules.d/40-redhat.rules /etc/udev/rules.d
     $ sudo sed -i 's/SUBSYSTEM!="memory", ACTION!="add", GOTO="memory_hotplug_end"/SUBSYSTEM=="*", GOTO="memory_hotplug_end"/' /etc/udev/rules.d/40-redhat.rules
     

   Reboot the system to initialize the above changes

4. Continue to installing NVIDIA Docker for RHEL

Ubuntu 18.04

Recommended

Follow the installation instructions on NVIDIA’s CUDA Installation Guide for full instructions suitable for your platform. The information listed below is a summary of the necessary steps, and does not cover the full range of options available.

1. Install the kernel headers and development tools necessary

   $ sudo apt-get update
   $ sudo apt-get install linux-headers-$(uname -r) gcc
   

2. Install the CUDA repository and driver on Ubuntu

   $ curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.243-1_amd64.deb
   $ sudo dpkg -i cuda-repo-ubuntu1804_10.1.243-1_amd64.deb
   $ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
   $ sudo apt-get update && sudo apt-get install -y nvidia-driver-418
   

3. Restart your machine

   sudo reboot

4. Verify the installation completed correctly, by asking nvidia-smi, NVIDIA’s system management interface application, to list the available GPUs.

   $ nvidia-smi -L
   GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-...)
   GPU 1: Tesla V100-PCIE-16GB (UUID: GPU-...)
   

5. Enable NVIDIA’s persistence daemon. This is mandatory for IBM POWER, but is recommended for other platforms as well.

   $ sudo systemctl enable nvidia-persistenced
   

6. Continue to installing NVIDIA Docker for Ubuntu

Install Docker CE and NVIDIA docker

CentOS 7 / RHEL 7 / Amazon Linux (x64)

Note

For IBM POWER9, see the next section installing NVIDIA Docker for IBM POWER

1. Follow the instructions for Docker CE for your platform at Get Docker Engine - Community for CentOS

2. Tell Docker to start after a reboot

   $ sudo systemctl enable docker && sudo systemctl start docker
   

3. Verify that docker is running

   $ sudo systemctl status docker
   ● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-08-12 08:22:30 IDT; 1 months 27 days ago
     Docs: https://docs.docker.com
     Main PID: 65794 (dockerd)
     Tasks: 76
   Memory: 124.5M
   CGroup: /system.slice/docker.service
           └─65794 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
   

4. Let your current user manage Docker, without requiring sudo

   $ sudo usermod -aG docker $USER
   

   Then, log out and log back in:

   $ exit
   

5. Install nvidia-docker

   $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
   $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
   $ sudo yum install -y nvidia-container-toolkit
   $ sudo yum install nvidia-docker2
   $ sudo pkill -SIGHUP dockerd
   $ sudo systemctl restart docker
   

   Note

   Occasionally, there may be a signature verification error while obtaining nvidia-docker. The error looks something like this: [Errno -1] repomd.xml signature could not be verified for nvidia-docker

   Run the following commands to update the repository keys:

   $ DIST=$(sed -n 's/releasever=//p' /etc/yum.conf)
   $ DIST=${DIST:-$(. /etc/os-release; echo $VERSION_ID)}
   $ sudo rpm -e gpg-pubkey-f796ecb0
   $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/*/gpgdir --delete-key f796ecb0
   $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/nvidia-docker/gpgdir --delete-key f796ecb0
   $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/nvidia-container-runtime/gpgdir --delete-key f796ecb0
   $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/libnvidia-container/gpgdir --delete-key f796ecb0
   $ sudo yum update
   

6. Verify the NVIDIA docker installation

   $ sudo docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi -L
   GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-...)
   GPU 1: Tesla V100-PCIE-16GB (UUID: GPU-...)
   

7. Continue to Installing the SQream DB Docker container

CentOS 7 / RHEL 7 (IBM POWER)

1. Install Docker for IBM POWER

   $ wget http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/docker-ppc64el/container-selinux-2.9-4.el7.noarch.rpm
   $ wget http://ftp.unicamp.br/pub/ppc64el/rhel/7_1/docker-ppc64el/docker-ce-18.03.1.ce-1.el7.centos.ppc64le.rpm
   $ yum install -y container-selinux-2.9-4.el7.noarch.rpm docker-ce-18.03.1.ce-1.el7.centos.ppc64le.rpm
   

2. Tell Docker to start after a reboot

   $ sudo systemctl enable docker && sudo systemctl start docker
   

3. Verify that docker is running

   $ sudo systemctl status docker
   ● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-08-12 08:22:30 IDT; 1 months 27 days ago
     Docs: https://docs.docker.com
     Main PID: 65794 (dockerd)
     Tasks: 76
   Memory: 124.5M
   CGroup: /system.slice/docker.service
           └─65794 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
   

4. Let your current user manage Docker, without requiring sudo

   $ sudo usermod -aG docker $USER
   

   Note

   Log out and log back in again after this action

5. Install nvidia-docker

   • Install the NVIDIA container and container runtime packages from NVIDIA’s repository:

     $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
     $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
     
     $ sudo yum install -y libnvidia-container* nvidia-container-runtime*
     

   • Add the NVIDIA runtime to the Docker daemon and restart docker:

     $ sudo mkdir -p /etc/systemd/system/docker.service.d/
     $ echo -e "[Service]\nExecStart\nExecStart=/usr/bin/dockerd --add-runtime=nvidia=/usr/bin/nvidia-container-runtime" | sudo tee /etc/systemd/system/docker.service.d/override.conf
     
     $ sudo systemctl daemon-reload && sudo systemctl restart docker
     

   Note

   Occasionally, there may be a signature verification error while obtaining nvidia-docker. The error looks something like this: [Errno -1] repomd.xml signature could not be verified for nvidia-docker

   Run the following commands to update the repository keys:

   $ DIST=$(sed -n 's/releasever=//p' /etc/yum.conf)
   $ DIST=${DIST:-$(. /etc/os-release; echo $VERSION_ID)}
   $ sudo rpm -e gpg-pubkey-f796ecb0
   $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/*/gpgdir --delete-key f796ecb0
   $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/nvidia-docker/gpgdir --delete-key f796ecb0
   $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/nvidia-container-runtime/gpgdir --delete-key f796ecb0
   $ sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/latest/libnvidia-container/gpgdir --delete-key f796ecb0
   $ sudo yum update
   

6. Verify the NVIDIA docker installation succeeded

   $ docker run --runtime=nvidia --rm nvidia/cuda-ppc64le:10.1-base nvidia-smi -L
   GPU 0: Tesla V100-SXM2-16GB (UUID: GPU-...)
   GPU 1: Tesla V100-SXM2-16GB (UUID: GPU-...)
   

7. Continue to Installing the SQream DB Docker container

Ubuntu 18.04 (x64)

1. Follow the instructions for Docker CE for your platform at Get Docker Engine - Community for CentOS

2. Tell Docker to start after a reboot

   $ sudo systemctl enable docker && sudo systemctl start docker
   

3. Verify that docker is running

   $ sudo systemctl status docker
   ● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-08-12 08:22:30 IDT; 1 months 27 days ago
     Docs: https://docs.docker.com
     Main PID: 65794 (dockerd)
     Tasks: 76
   Memory: 124.5M
   CGroup: /system.slice/docker.service
           └─65794 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
   

4. Let your current user manage Docker, without requiring sudo

   $ sudo usermod -aG docker $USER
   

   Note

   Log out and log back in again after this action

5. Install nvidia-docker

   $ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
   $ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
   $ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
   
   $ sudo apt-get update
   
   $ sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit nvidia-docker2
   $ sudo pkill -SIGHUP dockerd
   $ sudo systemctl restart docker
   

6. Verify the NVIDIA docker installation

   $ sudo docker run --runtime=nvidia --rm nvidia/cuda:10.1-base nvidia-smi -L
   GPU 0: Tesla V100-PCIE-16GB (UUID: GPU-...)
   GPU 1: Tesla V100-PCIE-16GB (UUID: GPU-...)
   

7. Continue to Installing the SQream DB Docker container

Preparing directories and mounts for SQream DB

SQream DB contains several directories that need to be defined

Directories and paths

Path name	Definition
storage	The location where SQream DB stores data, metadata, and logs
exposed path	A location that SQream DB can read and write to. Used for allowing access to shared raw files like CSVs on local or NFS drives
logs	Optional location for debug logs

Note

By default, SQream DB can’t access any OS path. You must explicitly allow it.

Install the SQream DB Docker container

1. Download the SQream DB tarball and license package

   In the e-mail from your account manager at SQream, you have received a download link for the SQream DB installer and a license package. Download the SQream DB tarball to the user home directory. For example:

   $ cd ~
   $ curl -O {download URL}
   

2. Extract the tarball into your home directory

   $ tar xf sqream_installer-2.0.7-DB2019.2.1.4-CO1.7.5-ED3.0.1-x86_64.tar.gz
   

3. Copy the license package

   Copy the license package from your home directory to the license subdirectory which is located in the newly created SQream installer directory.

   For example, if the licence package is titled license_package.tar.gz:

   $ cp ~/license_package.tar.gz sqream_installer-2.0.7-DB2019.2.1.4-CO1.7.5-ED3.0.1-x86_64/license
   

4. Enter the installer directory

   $ cd sqream_installer-2.0.7-DB2019.2.1.4-CO1.7.5-ED3.0.1-x86_64
   

5. Install SQream DB

   In most cases, the installation command will look like this:

   $ ./sqream-install -i -k -v <path to storage> -d <path to shared folder> -l <path to debug logs directory>
   

   For example, if the main storage path for SQream DB is /mnt/largedrive and the desired shared access path is /mnt/nfs/source_files, the command will look like:

   $ ./sqream-install -i -k -v /mnt/largedrive -d /mnt/nfs/source_files
   

   For a full list of options and commands, see the sqream-installer reference

6. SQream DB is now successfully installed, but not yet running.

Starting your first local cluster

1. Enter the sqream console utility, which helps coordinate SQream DB components

   $ ./sqream-console
   

2. Start the master components:

   sqream-console>sqream master --start
     starting master server in single_host mode ...
     sqream_single_host_master is up and listening on ports:   3105,3108
   

3. Start workers to join the cluster:

   sqream-console>sqream worker --start 2
     started sqream_single_host_worker_0 on port 5000, allocated gpu: 0
     started sqream_single_host_worker_1 on port 5001, allocated gpu: 1
   

   Note

   By default, each worker is allocated a full GPU. To launch more workers than available GPUs, see the sqream console reference

4. SQream DB is now running! (Exit the console by typing exit when done.)

What’s next?

• Test your SQream DB installation by creating your first table
• Connect an external tool to SQream DB
• Additional system configuration for performance and stability