Upgrading SlashDB 1.7.x to 1.8.x
Before upgrading, make sure you have a valid license key for version 1.8.x
Note that SlashDB 1.8.x supports Python versions 3.8 - 3.11
If you have set the SlashDB INI parameter slashdb_env
to a custom directory, replace any instances of /etc/slashdb
in this document with the custom directory.
General Upgrade Notes
Configuration Files Permissions
Files in the /etc/slashdb
folder should have the following permissions:
[root@8fc37589d913 /]# ls -l /etc/slashdb/
total 48
-rw-rw---- 1 root slashdb 2349 Jul 22 11:23 auth.cfg
-rw-rw---- 1 slashdb slashdb 382 Jul 22 11:23 databases.cfg
-rw-rw---- 1 slashdb slashdb 468 Jul 22 11:23 license.key
-rw-r--r-- 1 root root 4927 Jul 22 11:23 nginx.conf
-rw-rw---- 1 slashdb slashdb 4661 Jul 22 11:23 querydefs.cfg
-rw-r--r-- 1 slashdb slashdb 0 Jul 22 11:23 regular-initial
drwxr-xr-x 2 root root 4096 Aug 23 12:48 sites-available
drwxr-xr-x 2 root root 4096 Jul 22 11:23 sites-enabled
-rw-rw---- 1 root slashdb 6055 Jul 22 11:23 slashdb.ini
-rw-rw---- 1 slashdb slashdb 474 Jul 22 11:23 users.cfg
RPM / DEB Packages
You have two options to perform the upgrade:
- Installing over an existing installation
- can be faster for upgrading existing hosts
- manual edits of certain configuration files may be required
- Installing on a new system and copying the config files from the existing installation
- best for creating a new "golden image" or new client instance
- avoids stale components or application files whose locations might have changed
- reduces downtime
- does not affect production service in case any issues are encountered during the installation
Installing Over Existing Installation
1. Obtain and install the latest RPM/DEB on the system Create a snapshot image of the existing instance in case something goes wrong and the instance needs to be restored quickly.
Download the 1.8.x installer package:
- RPM https://downloads.slashdb.com/versions/1.8.90/slashdb-1.8.90.x86_64.rpm
- DEB https://downloads.slashdb.com/versions/1.8.90/slashdb_1.8.90_amd64.deb
(these links are for SlashDB v1.8.90. Replace the link with the specific version you want to download)
Red Hat Linux/RPM upgrade
cd /tmp
wget https://downloads.slashdb.com/versions/1.8.90/slashdb-1.8.90.x86_64.rpm
sudo su
yum upgrade ./slashdb-1.8.90.x86_64.rpm
Ubuntu/Debian/DEB upgrade
cd /tmp
wget https://downloads.slashdb.com/versions/1.8.90/slashdb_1.8.90_amd64.deb
sudo su
apt-get --only-upgrade install ./slashdb_1.8.90_amd64.deb
Install and configure an Oracle client, following these steps:
2. Review and update the slashdb.ini file as needed, as well as any other config files that were customized
Immediately after upgrading with the 1.8.x package, SlashDB will automatically start and use the existing configuration files from the v1.7.x instance.
The new instance will be in initial setup mode, which requires visiting the web UI and going through the welcome wizard:
- providing license details
- accepting terms & conditions
- setting the admin password
Alternatively, this can be done in the backend. After upgrading with the new package:
- remove file
/etc/slashdb/regular-initial
- enter a valid license in the file
/etc/slashdb/license.key
. The original license that was used in the 1.7.x deployment is backed up to a file with a datetime stamp in it e.g.license-202208241610.key
. - the configuration files below are automatically updated (there should be no need to edit these):
/etc/slashdb/users.cfg
/etc/slashdb/databases.cfg
/etc/slashdb/querydefs.cfg
/etc/slashdb/auth.cfg
-
the configuration files below may require manual review and editing:
/etc/slashdb/slashdb.ini
/etc/slashdb/nginx.conf
and- files in
/etc/slashdb/sites-available
and/etc/slashdb/sites-enabled
These files may be different in the newer version and won't be updated automatically. For these files, if a corresponding file with the suffix
.rpmnew
exists, then they must be compared using a tool likesdiff
and differences resolved manually.See section: Config File Differences Between 1.7.x and 1.8.x
-
If changes were made to any of the configuration files, restart the SlashDB service:
systemctl restart slashdb
Installing On New System and Copying Configuration From Existing Instance
1. Create backups of the SlashDB configuration files from the old system On the existing SlashDB 1.7.x instance:
- Create backups of the SlashDB configuration and sites directories:
cp /etc/slashdb ~/backups/v1.7.x/etc/slashdb
cp /var/lib/slashdb ~/backup/v1.7.x/var/lib/slashdb/sites
- Transfer these backups to the new system
2. Obtain and install the latest RPM/DEB on the new system On the new system:
-
perform normal installation of the v1.8.x package as per the documentation
-
install and configure an Oracle client, following these steps:
3. Overwrite the new system's SlashDB configuration files with the backups from the old system Transfer and replace some of the configuration files:
-
make a local backup of the configuration files of the new installation:
cp -r /etc/slashdb ~/backups/clean-v1.8.x
This will be needed to resolve differences between v1.7.x and v1.8.x in some of the configuration files.
-
remove the SlashDB configuration directory
/etc/slashdb
on the new machine:rm -r /etc/slashdb
-
Copy the 1.7.x instance backups to the
/etc/slashdb
and/var/lib/slashdb
directoriescp ~/backups/v1.7.x/etc/slashdb /etc/slashdb cp ~/backup/v1.7.x/var/lib/slashdb/sites /var/lib/slashdb
Important! Don’t copy or overwrite folders
/var/lib/slashdb/www
and/var/lib/slashdb/templates
. These contain web UI files and must match the installed version. -
update
/etc/slashdb/license.key
with a valid 1.8.x license -
the configuration files below are automatically updated (there should be no need to edit these):
/etc/slashdb/users.cfg
/etc/slashdb/databases.cfg
/etc/slashdb/querydefs.cfg
/etc/slashdb/auth.cfg
4. Review and update the slashdb.ini file as needed, as well as any other config files that were customized The configuration files below may require manual review and editing:
/etc/slashdb/slashdb.ini
/etc/slashdb/nginx.conf
and- files in
/etc/slashdb/sites-available
and/et/slashdb/sites-enabled
These files may be different in the newer version and won't be updated automatically.
They must be compared with files in ~/backups/clean-v1.8.x
using a tool like sdiff
and differences resolved manually.
See section: Config File Differences Between 1.7.x and 1.8.x
5. Reassign the IP address/hostname to the new system and shut down the old one
- resassign IP address/hostname to the new system
- restart the SlashDB service:
systemctl restart slashdb
Docker
To migrate a SlashDB 1.7.x Docker container to SlashDB 1.8.x, the simplest approach is to move the Docker volumes containing configuration data and logs from the 1.7.x container to the new 1.8.x container.
The SlashDB Docker containers include volumes for /etc/slashdb
and /var/log/slashdb
.
1. Obtain the latest SlashDB Docker image and config file bundle
First, download the SlashDB 1.8.x Docker image:
docker pull slashdb/slashdb
Next, download the default SlashDB configuration files and create a directory for logs
wget -c https://downloads.slashdb.com/versions/1.8.90/default-slashdb-configs_1.8.90.zip
unzip ./default-slashdb-configs_1.8.90.zip
mkdir slashdb-log
In your current directory you should have the zip file, and two directories named slashdb
and slashdb-log
.
There are a few changes in the SlashDB 1.8.x INI that you may want to keep. If so, make a reference copy of the slashdb/slashdb.ini
file before proceeding.
2. Copy existing configuration files from the old Docker container to the new one
Your existing 1.7.x container should have been configured on installation to store its configuration and logs in two directories named slashdb
and slashdb-log
. Locate these directories, and copy their contents to the newly created slashdb
and slashdb-log
directories, respectively.
3. Review and update the slashdb.ini file as needed, as well as any other config files that were customized
If you made a reference copy of the slashdb/slashdb.ini
file for the 1.8.x container, copy the changes you want from the reference file to slashdb/slashdb.ini
file. You may want to use a tool like sdiff
to resolve any differences. See Config File Differences Between 1.7.x and 1.8.x
You can now update the slashdb/license.key
file for the 1.8.x container with a valid 1.8.x license using a text editor. If you want to upload a new license via the GUI, navigate to the 1.8.x SlashDB GUI in a browser to upload a license file after the container is running.
4. Launch the new container
Start the SlashDB 1.8.x container:
docker run -d -p 8080:80 -v $(pwd)/slashdb:/etc/slashdb -v $(pwd)slashdb-log:/var/log/slashdb slashdb/slashdb
Your previous SlashDB configuration should be present on the 1.8.x container.
See the Docker Volumes documentation for more information about Docker volumes.
VMWare/VirtualBox/HyperV
1. Obtain and launch the latest virtual appliance
Download the latest virtual appliance: https://downloads.slashdb.com/versions/1.8.90/slashdb-1.8.90.ova
Install the appliance using the virtual machine management software for your virtual environment.
2. Copy the SlashDB configuration files from the old appliance to the new one
Launch the SlashDB 1.7.x appliance if it is not running, and create backups of /etc/slashdb/
and /var/log/slashdb
from a shell:
systemctl stop slashdb
tar cvf etc_slashdb.tar /etc/slashdb
tar cvf var_log_slashdb.tar /var/log/slashdb
Launch the SlashDB 1.8.x appliance if it is not running. You may need to update the network settings first to configure the appliance for your hardware/network environment.
There are a few changes in the SlashDB 1.8.x INI that you may want to keep. If so, make a reference copy of the /etc/slashdb/slashdb.ini
file on the 1.8.x appliance before proceeding.
Copy the tar
backup files from the 1.7.x appliance to the 1.8.x appliance using SCP, FTP, a shared network resource, USB, etc.
Using SCP to copy the tar files from the 1.7.x appliance (default password for root on the SlashDB 1.8.x appliance is slashdb
):
scp etc_slashdb.tar root@<slashdb_17x_appliance_hostname_ip>:
scp var_log_slashdb.tar root@<slashdb_17x_appliance_hostname_ip>:
Open a shell on the 1.8.x appliance. The tar
files will be in the /root
folder. You'll need to install the tar
utility to extract them:
yum install tar
Extract the backups to their target directories:
cd /etc && tar xvf /root/etc_slashdb.tar --strip 1
cd /var/log && tar xvf /root/var_log_slashdb.tar --strip 1
3. Review and update the slashdb.ini file as needed, as well as any other config files that were customized
If you made a reference copy of the /etc/slashdb/slashdb.ini
file on the 1.8.x appliance, copy the changes you want from the reference file to the /etc/slashdb/slashdb.ini
file. You may want to use a tool like sdiff
to resolve any differences. See Config File Differences Between 1.7.x and 1.8.x for more details.
Restart the SlashDB process:
systemctl restart slashdb-app
4. Reassign the IP address/hostname to the new appliance and shut down the old one
You can now update the /etc/slashdb/license.key
file on the 1.8.x appliance with a valid 1.8.x license using a text editor, or navigate to the 1.8.x SlashDB GUI in a browser to upload a license file.
Amazon AWS/Microsoft Azure
Upgrades for Amazon AWS/Microsoft Azure deployments follow a process similar to upgrading virtual appliances. Once the new instance is deployed, you can reassign the IP address/hostname of the old deployment to the new one.
1. Launch the new instance from AWS/Azure Marketplace following normal installation instructions
2. Copy and reconcile the SlashDB configuration files from the existing instance to the new one
From the SlashDB 1.7.x container, create backups of the configuration:
systemctl stop slashdb
tar cvf etc_slashdb.tar /etc/slashdb
tar cvf var_log_slashdb.tar /var/log/slashdb
There are a few changes in the SlashDB 1.8.x INI that you may want to keep. If so, make a reference copy of the /etc/slashdb/slashdb.ini
file after launching the AWS/Azure SlashDB 1.8.x container.
Copy the tar
backup files from the 1.7.x container to the 1.8.x container using SCP, FTP, or a shared network resource.
Using SCP to copy the tar files from the 1.7.x container:
scp etc_slashdb.tar root@<slashdb_17x_container_hostname_ip>:
scp var_log_slashdb.tar root@<slashdb_17x_container_hostname_ip>:
Open a shell on the 1.8.x container. The tar
files will be in the /root
folder.
Extract the backups to their target directories:
cd /etc && tar xvf /root/etc_slashdb.tar --strip 1
cd /var/log && tar xvf /root/var_log_slashdb.tar --strip 1
3. Reassign the IP address/hostname to the new instance and shut down the old one
If you made a reference copy of the /etc/slashdb/slashdb.ini
file on the 1.8.x container, copy the changes you want from the reference file to the /etc/slashdb/slashdb.ini
file. You may want to use a tool like sdiff
to resolve any differences. See Config File Differences Between 1.7.x and 1.8.x for more details.
Finally, restart the SlashDB process:
systemctl restart slashdb-app
Config File Differences Between 1.7.x and 1.8.x
Changes to SlashDB INI configuration file
The changes below refer to the /etc/slashdb/slashdb.ini
file.
Removed a setting that was always set to default
-ignore_default_fields = False
Added titles to some of the settings and spacing for readability
+# Session secret
# IMPORTANT: Please generate a new secret key for this SlashDB instance.
# Instances running in a cluster should have the same secret key.
# Keep the secret key used in production secret!
# Memcached Server Settings
;memcached_server = 127.0.0.1:11211
;memcache_root_key = slashdb
+
# name of SlashDB memcache lock key
;memcache_lock_key = slashdb_registry_lock
+
# max time in seconds to wait before trying to access memcache when lock exists
;memcache_lock_sleep_interval = 0.1
Added new setting secret
for encrypting database credentials
+# Database credentials secret
+# IMPORTANT: Please generate a new secret key for this SlashDB instance.
+# The secret should be 32 alphanumeric characters or 32 bytes encoded base64.
+# Instances running in a cluster should have the same secret key.
+# Keep the secret key used in production secret!
+;secret =
+
Removed deprecated XSD_SDB Settings
-# XSD_SDB Settings
-xsd_sdb.root_tag = SlashDB
-xsd_sdb.xmlns = http://www.vtenterprise.com/slashdb
-
Disabled the CORS configuration which by default allowed traffic from any origin.
The section [pipeline:main]
was changed:
[pipeline:main]
-pipeline = cors sdbapp
+# Please uncomment the line "; cors" to enable CORS policies
+# for web applications from other origins.
+pipeline =
+; cors
+ sdbapp
Changed the comment for the CORS "free" policy
[filter:cors]
use = egg:wsgicors#middleware
# CORS available policies
-# "free" policy
+# "free" policy allows traffic from any origin
free_origin = copy
free_headers = *
free_expose_headers = Set-Cookie, Warning
Removed sqlalchemy
from section [loggers]
This should only be used for debugging; the commented line can be appended to keys
if needed.
[loggers]
-keys = root, slashdb, exc_logger, sqlalchemy
-# sqlachemy_engine, sqlachemy_pool
+keys = root, slashdb, exc_logger
+# sqlalchemy, sqlalchemy_engine, sqlalchemy_pool
Changed propagation of logger exc_logger
This prevents exception log messages from being duplicated by the root
logger. The exc_logger
saves logs to its own file.
[logger_exc_logger]
level = ERROR
handlers = exc_handler
qualname = exc_logger
-propagate = 1
+propagate = 0
Replaced variable log_file
with path to file in args
[handler_exc_handler]
-log_file = exceptions.log
class = handlers.RotatingFileHandler
-args = ("/var/log/slashdb/%(log_file)s",'a', 2*1024*1024, 5, 'utf-8')
+args = ("/var/log/slashdb/exceptions.log",'a', 2*1024*1024, 5, 'utf-8')
level = ERROR
formatter = exc_formatter
Updated application log settings to create a unified format
-[formatter_exc_formatter]
-format = %(asctime)s %(message)s
+[formatter_generic]
+format = %(asctime)-.19s | %(created)d%(msecs)03d | sdbapp | %(name)s | %(levelname)-5.5s | %(process)d | %(threadName)s | | | | | | | | | | | %(module)s:%(funcName)s:%(lineno)d | | %(message)s
[formatter_slashdb]
-format = %(asctime)-.19s | %(created)d%(msecs)03d | sdbapp | %(levelname)-5.5s | %(processName)s | %(threadName)s | %(name)s | %(module)s:%(funcName)s:%(lineno)d | %(user_id)s | %(message)s | pid: %(process)d
+format = %(asctime)-.19s | %(created)d%(msecs)03d | sdbapp | %(name)s | %(levelname)-5.5s | %(process)d | %(threadName)s | %(request_id)s | %(request_method)s | %(request_url)s | %(response_status_code)s | | | | %(request_client_addr)s | %(request_referer)s | %(request_user_agent)s | %(module)s:%(funcName)s:%(lineno)d | %(user_id)s | %(message)s
-[formatter_generic]
-format = %(asctime)-.19s | %(created)d%(msecs)03d | sdbapp | %(levelname)-5.5s | %(processName)s | %(threadName)s | %(name)s | %(module)s:%(funcName)s:%(lineno)d | | %(message)s | pid: %(process)d
+[formatter_exc_formatter]
+format = %(asctime)-.19s | %(created)d%(msecs)03d | sdbapp | %(name)s | %(levelname)-5.5s | %(process)d | %(threadName)s | | | | | | | | | | | %(module)s:%(funcName)s:%(lineno)d | | %(message)s
# End of Logging Settings
Changed listen
queue to 0
Unhandled requests will be reported more quickly.
auto-procname = true
procname-prefix-spaced = slashdb
master-fifo = /run/slashdb/master-fifo
-listen = 100
+listen = 0
Updated uWSGI log settings in [uwsgi]
section to create a unified log format
# uWSGI Logging Settings
logto = /var/log/slashdb/uwsgi.log
logfile-chown = slashdb:slashdb
# 20 MB
log-maxsize = %(20 * 1024 * 1024)
-log-encoder = format ${strftime:%%Y-%%m-%%d %%H:%%M:%%S} | ${millis} | uwsgi | INFO | | | | | | ${msgnl}
-# requests-logging
-log-format = "%(method) %(uri) %(proto)" | pid: %(pid) wid: %(wid) | %(addr) | %(status) | %(secs) s | | resp %(size) bytes | ipktsize: %(pktsize) bytes | reqbody: %(cl) bytes | %(headers) headers | in %(hsize) bytes | %(switches) switches | on core %(core)
+
+# internal uWSGI logs
+logger = stdio
+log-encoder = format ${strftime:%%Y-%%m-%%d %%H:%%M:%%S} | ${millis} | uwsgi | uwsgi | INFO | | | | | | | | | | | | | | | ${msgnl}
+
+# request logs
+logger-req = stdio
+log-req-encoder = format ${strftime:%%Y-%%m-%%d %%H:%%M:%%S} | ${millis} | uwsgi | uwsgi | INFO | ${msgnl}
+log-format = %(pid) | uWSGIWorker%(wid)Core%(core) | %(var.REQUEST_ID) | %(method) | %(uri) | %(status) | %(secs) | | %(size) | %(addr) | %(referer) | %(uagent) | | |
Changes to NGINX configuration file
The changes below refer to the /etc/slashdb/nginx.conf
file.
Updated settings for SSL to increase NGINX security
##
# SSL Settings
+ # https://ssl-config.mozilla.org
##
+ ssl_session_timeout 1d;
+ ssl_session_cache shared:MozSSL:10m; # about 40000 sessions
+ ssl_session_tickets off;
+
ssl_protocols TLSv1.2 TLSv1.3;
- ssl_prefer_server_ciphers on;
+ ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS;
+ ssl_prefer_server_ciphers off;
Changed NGINX access log format to create a unified log format.
map $msec $msec_int {
"~(.+)\.(.+)" "$1$2";
}
- log_format sdblog '$time_iso8601mod | $msec_int | nginx | INFO '
- ' | | | | | | "$request" | pid: $pid | $remote_addr '
- '| $status | $request_time s | req $request_length bytes'
- ' | resp $bytes_sent bytes | | | | | | | "$http_referer" | '
- '"$http_user_agent"';
+
+ log_format sdblog '$time_iso8601mod | $msec_int | nginx | nginx | INFO '
+ '| $pid | | $request_id | $request_method | $request_uri '
+ '| $status | $request_time | $request_length | $bytes_sent '
+ '| $remote_addr | $http_referer | $http_user_agent | | | ';
access_log /var/log/nginx/access.log sdblog;
error_log /var/log/nginx/error.log;
Fixed indentation in retry mapping
# retry-after 2s for /login
map $status $retry_after {
- default '';
- 429 '2';
+ default '';
+ 429 '2';
}
Added a commented out section for redirecting HTTP to HTTPS
+ # Redirect HTTP to HTTPS
+ # server {
+ # listen 80 default_server;
+ # listen [::]:80 default_server;
+ # location / {return 301 https://$host$request_uri;}
+ # }
+
Fixed indentation in comments
# server_name your-slashdb.com;
listen 80;
- # listen 443 ssl;
- # ssl_certificate /pah/to/certificate;
- # ssl_certificate_key /path/to/key;
+ # listen 443 ssl;
+ # ssl_certificate /pah/to/certificate;
+ # ssl_certificate_key /path/to/key;
Moved headers to outer scope
Moved Retry-After
and Cache-Control
headers to the config outer scope so that the security headers will be added correctly to the responses. See the related item Removed headers from location.
send_timeout 5m;
charset utf-8;
+ add_header Retry-After $retry_after always;
+ add_header Cache-Control "no-cache, private" always;
+
Updated security headers
# Secure headers
- add_header X-Frame-Options DENY;
- add_header X-Content-Type-Options nosniff;
- add_header Referrer-Policy same-origin;
+ add_header X-Frame-Options DENY always;
+ add_header X-Content-Type-Options nosniff always;
+ add_header Referrer-Policy same-origin always;
add_header Content-Security-Policy "default-src 'self' www.youtube.com *.microsoftonline.com *.google.com *.okta.com *.onelogin.com; style-src 'self' 'unsafe-inline';" always;
-
+ add_header Strict-Transport-Security "max-age=63072000" always;
+ add_header Permissions-Policy "fullscreen=(self \"https://www.youtube.com\"), clipboard-read=(self), clipboard-write=(self)" always;
+ # add_header Cross-Origin-Embedder-Policy require-corp always;
+ add_header Cross-Origin-Opener-Policy same-origin always;
+ add_header Cross-Origin-Resource-Policy same-origin always;
+
Removed blank spaces between option name and value.
location ~ /db*|/query* {
- uwsgi_pass slashdb_app;
- include uwsgi_params;
+ uwsgi_pass slashdb_app;
+ include uwsgi_params;
+ uwsgi_param REQUEST_ID $request_id;
# Uncomment the next line to enable uwsgi cache for this location
#uwsgi_cache SDB_CACHE;
- uwsgi_cache_key $host$request_uri$http_authorization$cookie_auth_tkt$api_key_get$api_key_header;
- uwsgi_cache_min_uses 1;
- uwsgi_cache_use_stale error timeout invalid_header http_500;
+ uwsgi_cache_key $host$request_uri$http_authorization$cookie_auth_tkt$api_key_get$api_key_header;
+ uwsgi_cache_min_uses 1;
+ uwsgi_cache_use_stale error timeout invalid_header http_500;
}
Added request id to uwsgi_param
This passes the request ID to SlashDB. The request ID is used by in logging to track the same request across NGINX, uWSGI and SlashDB.
location / {
uwsgi_pass slashdb_app;
- include uwsgi_params;
+ include uwsgi_params;
+ uwsgi_param REQUEST_ID $request_id;
Removed headers from location
Headers Retry-After
and Cache-Control
were moved to the config outer scope so that the security headers will be added correctly to the responses. See the related item Moved headers to outer scope.
location /login {
# allow up to 5 requests in queue, then reject
limit_req zone=login_limit burst=5 nodelay;
uwsgi_pass slashdb_app;
- add_header Retry-After $retry_after always;
}
}
location ^~ /static/ {
alias /var/lib/slashdb/www/;
- add_header Cache-Control "no-cache, private";
}
}