SAVIME: Main Page

Introduction

SAVIME (Simulation Analysis and Visualization In MEmory) is a DBMS-like system that stores data in the format of typed multidimensional arrays. It is desgined to help querying and vizualize numerical simulation data. However, it can also be used as an analytical in-memory array database management system. SAVIME is at early development stages, and supports only a handful of operations. But it is already able to perform many tasks, even for large datasets in memory.

Compiling and Running

To install SAVIME, you need to clone the git repository and compile it in your environment.

Prerequesites

The following packages must be installed for successfully compiling and running SAVIME:

sudo apt install git
			sudo apt install cmake
			sudo apt install libboost-dev
                        sudo apt install libboost-regex-dev
                        sudo apt install libboost-iostreams-dev
			sudo apt install libcurl4-openssl-dev
			sudo apt install libjsoncpp-dev

It is also advisable to install other packages depending on intended usage. In order to have a better performance in some operations, SAVIME uses TBB, which can be installed as follows:

sudo apt-get install libtbb-dev

If it is necessary to change the base of SAVIME grammar, then, the following packages (bison and flex) are required.

sudo apt install bison
			sudo apt install flex

If the experimental RDMA support is required, then it is necessary to install the following package:

 sudo apt install librdmacm-dev

Compiling and running SAVIME

Once you have all dependencies are installed, clone the repository:
git clone https://github.com/hllustosa/Savime

Create a separate build directory, configure and build SAVIME:

 mkdir savime-build
		cd savime-build
		cmake ${SAVIME_BUILD} -DCMAKE_BUILD_TYPE=Release
		make -j 4
		sudo make install

Where ${SAVIME_BUILD} is the path where the SAVIME repository is in your environment.

Once it is done, you can Run the SAVIME server by typing:

 savime

You can also call the savime client with your desired query by typing:

 savimec 'your query;'

You can test savime by running these commands from savime build directory:

 cd ${SAVIME_BUILD}/examples
		savime -D
		./test_savime.sh

Where the option -D makes Savime Run as a daemon. You should see the outputs of various samples of queries in teste_savime.sh file. You can finish Savime by running:

 savimec -k

Compiling and running SAVIME with catalyst support

To build and running SAVIME with catalyst support, it is first necessary to have built Paraview. To do so, first install these dependencies:

 sudo apt install cmake-curses-gui
		sudo apt-get install freeglut3-dev

Clone Paraview's repository and update the submodules:

 git clone https://gitlab.kitware.com/paraview/paraview.git
		cd paraview
		git submodule update --init --recursive
		git pull
		git submodule update

Now, create a separated build directory and configure Paraview. While configuring, don't forget to set the following parameters: PARAVIEW_USE_MPI=off, PARAVIEW_ENABLE_PYTHON=on, PARAVIEW_BUILD_QT=off and PARAVIEW_INSTALL_DEVELOPMENT_F=on. For more info, check the Paraview's Documentation.

 mkdir paraview_build
		cd paraview_build
		ccmake ../paraview

Compile Paraview:

 make -j 4
		sudo make install

VTK library files for catalyst are installed in /usr/local/lib. You should add this to your PATH variable. To do this for a single session, Run:

 export LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib

Or add /usr/local/lib to /etc/ld.so.conf to make the changes permanent. Once you finished this, you should compile SAVIME set configure parameter to enable catalyst. Like this:

 mkdir savime-build
		cd savime-build
		cmake ${SAVIME_BUILD} -DCMAKE_BUILD_TYPE=Release -DBUILD_WITH_CATALYST=yes
		make -j 4
		sudo make install

Finally, you can test it by running these commands from savime build directory:

 cd examples
		savime -D -b ${SAVIME_BUILD}/etc/savime.boot
		./test_savime_catalyst.sh

After running it for a while you should see a bunch of PVTU files in examples/viz directory. They contain a visualization for a numerical simulation compatible with Paraview.

Operators

The following operators shown here are currently supported by SAVIME. We start describing the DDL operators, the are responsible for creating and removing the basic structures in SAVIME, such as a TAR or a DATASET. In SAVIME, data is ingested into DATASETS, and then these DATASETs are attached to SUBTAR in a given TAR via the load_subtar command. Thefore, the main operators are:

CREATE_TAR

CREATE_TAR("tar_name", "type_name", "(dimensions_block)", "(attributes_block)", ["(roles_block)"]);

CREATE_TYPE

CREATE_TYPE("[type_name](mandatory_role1, mandatory_role2, *non_mandatory_role3, ..., roleN)");

CREATE_DATASET

CREATE_DATASET("name:type", "data_file_path");
			CREATE_DATASET("name:type", "literal dataset");
			CREATE_DATASET("name:type", "initial:spacing:final");

LOAD_SUBTAR

LOAD_SUBTAR("tar_name", "dimension_specs", "dataset_specs");

DROP_*

DROP_TAR("tar_name");
			DROP_TYPE("type_name");
			DROP_DATASET("dataset_name");

SAVIME supports a range of very common DML operator that when combine can create the most varied queries. These DML operators are:

SCAN

Executes query and iterates through the result without sending it to the client. Used for measuring query execution time. SCAN(tar);

SELECT

Projects a subset of dimensions and attributes of a TAR. SELECT(tar, data_element1, data_element2, ..., data_element_n);

WHERE

Filters data according to a complex logical predicate. WHERE(tar, logical_predicate);

SUBSET

Clips a TAR by limiting a range of values for its dimensions. Possibly more efficient than WHERE in most cases. SUBSET(tar, dim1, lower_bound1, upper_bound1, lower_bound2, upper_bound2, ..., lower_bound_n, upper_bound_n);

DERIVE

Creates a new derived attribute by calculating an arithmetic expressions involving indexes values. attributes, constants, arithmetic operations and functions. DERIVE(tar, new_attribute_name, arithmetic expression);

CROSS

Creates the cartesion products of cells in the left and right array. The resulting array has a number of dimensions and attributes equal to the sum of dimensions in both left and right TARs. CROSS(left_tar, right_tar);

DIMJOIN

Creates a TAR as the junction of attributes between two tars by matching pairs of dimensions.

DIMJOIN(left_tar, right_tar, left_tar_dim1, right_tar_dim1, left_tar_dim2, right_tar_dim2, ..., left_tar_dim_n, right_tar_dim_n, );

AGGREGATE

Group attribute and dimension values by dimensions and calculate aggregation functions.

AGGREGATE(tar, agg_func1, data_element1, new_attribute1, ..., agg_func_n, data_element_n, new_attribute_n, dimension1, dimension2, ..., dimension_n );

Some special operator complete the set of possibilites currently available in SAVIME. They are:

STORE

Maintains the result of a query into the system, thus making it available for futher analysis. STORE(savime_dml_query, "new_tar_name");

CATALYZE

Create files for being vizualized in PARAVIEW or call CATALYST scripts for analysis. CATALYZE(field_data_tar1, [geometry_tar], [topology_tar], [catalyst_script], output_dir)