Data Manipulation Tools

• sddsbreak -- Breaks data pages into new, separate pages based on changes in column data and other criterion. Example applications: reorganizing a file to have a limited number of rows in each page, or to have a new page started when a gap is seen in the data.

• sddscollapse -- Collapses a data set into a single data page by deleting the tabular data and turning the parameters into columns. Example application: abstraction of summary properties of data set following analysis with sddsprocess.

• sddscollect -- Reorganizes tabular data from the input file to bring data from several groups of similarly named columns together into a single column per group. Example application: collecting several statistical analyses of many columns into a single column per analysis type.

• sddscombine -- Combines any number of data sets into a single data set by adding data from each successive data set to a newly-created data set. Example application: bringing together comparable but distinct data for analysis with sddsprocess. Using sddsprocess, sddscombine, and sddscollapse in sequence repeatedly is a powerful way to analyze and collate large amounts of data.

• sddsconvert -- Allows conversion of a data set between binary and ASCII, with optional deletion and renaming of columns, arrays, and parameters . Example application: conversion to binary of an ASCII data set created by a simple program, or by a text editor.

• sddsderef -- Allows dereferencing (i.e., de-indexing) of array and column data. Example application: converting a column of integer indexes into a column of equivalent text messages, where the text messages are stored once each in an array in the input file (for space-savings).

• sddsendian -- Converts from little-endian to big-endian and vice-versa. Example application: converting binary data from the native format to a format used on another type of computer prior to transferring the data to the other computer.

• sddsexpand -- Expands a data set into one page for each row, with column data promoted to parameter data. Essentially the inverse of sddscollapse.

• sddsregroup -- Swaps the row indexing and page indexing of data in an SDDS file. That is, the row of the data page in the input file becomes the row of the data page of the output file. Example application: viewing the long-term evolution of a repeatedly-sampled waveform at each point in the waveform.

• sddstranspose -- Transposes the tabular data in the input file, so that the output file contains one column for each row in the input. Example usage: tranpose an orbit response matrix as part of preparing to use it for feedback.

• sddsmatrixmult -- Multiplies the tabular data in the two input files to produce a file containing a matrix of the product. Example usage: Multiply a vector of errors with a correction matrix to obtain a vector of corrections to apply in a step-by-step feedback system.

• sddsselect -- Copies rows from one file based on the presence or absence of matching data in another file. Example application: finding all of the rows from one file that do not appear in a second file.

• sddssort -- Sorts the tabular data section of a data set by the values in named columns. Optionally eliminates duplicate rows.

• sddssplit -- Places each page of a file in a separate, new file. Example application: getting selected pages of a file into separate, single-page files for use with a program that only recognizes the first page.

• sddsxref -- Creates a new data set by adding selected rows from one data set to another data set. Example application: cross-referencing the turn-by-turn coordinates of particles in a tracking simulation with the initial coordinates using a particle ID number.