Release Notes 4.7

The 4.7 release notes were released on September 01, 2024

Compatibility Matrix

System Requirement

Details

Supported OS

RHEL 8.x

supported Nvidia driver

CUDA version 11.x

Storage version

51

Driver compatibility

  • JDBC 5.3.1

  • ODBC 4.4.4

  • NodeJS 4.2.4

  • .NET 5.0.0

  • Pysqream 5.2.0

  • Spark 5.0.0

  • SQLoader As A Service 8.1 (compatible with v4.6.1 or later)

  • SQLoader As A Process 7.13 (compatible with v4.5.13 or later)

New Features and Enhancements

► Enhance observability and enable shorter investigation times with the new Health Monitoring SQreamDB service.

► SQreamDB may now be deployed on AWS private cloud.

► A SUPERUSER may now release a specific lock or all locks blocking file cleanup and preventing operations on locked objects within the system.

► SQreamDB operates with utmost efficiency when processing tables containing large data chunks. Introducing a new RECHUNK utility function, it simplifies the management of tables with small data chunks. This feature enables users to merge small data chunks into larger ones and simultaneously eliminating any deleted records present.

► Enable automatic termination of queries that exceed a pre-defined time limit in the queue. The introduction of the queueTimeoutMinutes flag empowers you to set time constraints for queries in the queue, ranging from a few minutes to a maximum of 72 hours.

► Safely cast data types with the new IsCastable function. This function allows you to check whether a cast operation is possible or supported for a given column and data type and provides an alternative when an exception occurs when used within a CASE statement.

► JDBC enhancements have been implemented to facilitate the retrieval of the record count for the updated number of rows during INSERT and DELETE operations when connecting to a third-party platform via JDBC. Use the SQreamDB JDBC connector as usual; the sole distinction is in the ability to now observe the updated number of rows.

► Enhance your COPY FROM operations with the new DELETE_SOURCE_ON_SUCCESS parameter, which automatically deletes the source file being copied into SQreamDB. This not only saves time and effort in cleaning storage but also helps conserve storage space.

► You may now retrieve and manipulate data from different databases within a single SQreamDB cluster through the execution of a single SQL statement using the Cross-Database syntax.

► You may now use the new logFormat flag to configure the format of your log files. By default, logs are saved as CSV files. To configure your log files to be saved as JSON instead, use the logFormat flag in your legacy config file. If your current logs are in CSV format and you require Health Monitoring, it’s advisable to configure your logs to be saved in both CSV and JSON formats as outlined above.

Note

The logFormat flag must be configured identically in both your legacy_config_file and your metadata_config_file

► You now have the option to choose the location for your metadata_server, server_picker, and Worker log files. In previous SQreamDB versions, the location of your log files was predetermined and hard-coded.

metadata_server

  • Using the metadata_server_config.json file:

    {
"logPath": "<home/logs/my_logs>"
}

  • Using the CLI:

    ./metadata_server --log_path=<home/logs/my_logs>

Server Picker

Using the CLI:

./server_picker --log_path=<home/logs/my_logs>

Worker

Using the sqream_config_legacy.json:

{
"DefaultPathToLogs": "<home/logs/my_logs>"
}

► For any new SQreamDB installation or upgrade, your default legacy configuration file will include the following cluster flags:

{
 "logMaxFileSizeMB": 20,
 "logFileRotateTimeFrequency": "daily",
}

Note

Starting with SQreamDB version 4.6, log file naming conventions have changed. Ensure that any code referencing log file names is updated accordingly.

  • When using the logFileRotateTimeFrequency flag, log file names will follow these patterns:

    Daily: sqream_yyyyMMdd_000.log

    Weekly: sqream_yyyyMMWW_000.log (WW = week number within the month)

    Monthly: sqream_yyyyMM_000.log

  • When using the logMaxFileSizeMB flag, log files will follow the pattern:

    sqream_N.log (N = 1 to 13)

Known Issues

Percentile is not supported for Window Functions

Version 4.7 resolved Issues

SQ No.

Description

SQ-15691

Fixed TEXT casting into DOUBLE and NUMERIC issue when using scientific notation

SQ-16038

Fixed CREATE TABLE.. LIKE permission heritage issue

SQ-16937

Fixed schema corruption following default permission altering issue

SQ-17149

Created a new SWAP_TABLE_NAMES utility function to address issue with views affected by SQLoader loads

SQ-17270

Enhanced orphan snapshot cleaning mechanism

SQ-17520

Fixed a SQLoader cleanup_extents related issue

SQ-17944

Fixed UNION query result issue

Deprecations

► Haskell CLI

Starting February 2025, support for the Haskell CLI will be discontinued, and it will be replaced by a JAVA CLI that is compatible with both SQreamDB.

► CentOS Linux 7.x

CentOS Linux 7.x has reached its end of life and is not supported by SQreamDB.

  • REHL 8.x is now officially supported.

Upgrading to Version 4.7

  1. Generate a back-up of the metadata by running the following command:

    select backup_metadata('out_path');

    Tip

    SQreamDB recommends storing the generated back-up locally in case needed.

    SQreamDB runs the Garbage Collector and creates a clean backup tarball package.

  2. Shut down all SQreamDB services.

  3. Copy the recently created back-up file.

  4. Replace your current metadata with the metadata you stored in the back-up file.

  5. Navigate to the new SQreamDB package bin folder.

  6. Run the following command:

    ./upgrade_storage <levelDB path>

  7. Version 4.4 introduces a service permission feature that enables superusers to grant and revoke role access to services. However, when upgrading from version 4.2 or earlier to version 4.4 or later, this feature initializes access to services, causing existing roles to lose their access to services.

There are two methods of granting back access to services:

Note

Upgrading from a major version to another major version requires you to follow the Upgrade Storage step. This is described in Step 7 of the Upgrading SQreamDB Version procedure.