Data Manipulation Tools

• sddsbreak -- Breaks data pages into new, separate pages based on changes in column data and other criterion. Example applications: reorganizing a file to have a limited number of rows in each page, or to have a new page started when a gap is seen in the data.

• sddscast -- change the data type of the elements in an SDDS file.

• sddscollapse -- Collapses a data set into a single data page by deleting the tabular data and turning the parameters into columns. Example application: abstraction of summary properties of data set following analysis with sddsprocess.

• sddscollect -- Reorganizes tabular data from the input file to bring data from several groups of similarly named columns together into a single column per group. Example application: collecting several statistical analyses of many columns into a single column per analysis type.

• sddscombine -- Combines any number of data sets into a single data set by adding data from each successive data set to a newly-created data set. Example application: bringing together comparable but distinct data for analysis with sddsprocess. Using sddsprocess, sddscombine, and sddscollapse in sequence repeatedly is a powerful way to analyze and collate large amounts of data.

• sddsconvert -- Allows conversion of a data set between binary and ASCII, with optional deletion and renaming of columns, arrays, and parameters . Example application: conversion to binary of an ASCII data set created by a simple program, or by a text editor. N.B.: it is not recommended to use sddsconvert to convert a binary SDDS file to ASCII, then strip the header off and read the ASCII file. This completely bypasses the self-describing aspects of the SDDS file and is not robust. If the program that creates the SDDS file is changed so that the columns are created in a different order, the program that reads the ASCII file will produce unexpected results. Use sdds2plaindata, sddsprintout, or sdds2stream for conversion to non-self-describing files. In this way, you can assure the order of the data is fixed.

• sddsderef -- Allows dereferencing (i.e., de-indexing) of array and column data. Example application: converting a column of integer indexes into a column of equivalent text messages, where the text messages are stored once each in an array in the input file (for space-savings).

• sddsendian -- Converts from little-endian to big-endian and vice-versa. Example application: converting binary data from the native format to a format used on another type of computer prior to transferring the data to the other computer.

• sddsexpand -- Expands a data set into one page for each row, with column data promoted to parameter data. Essentially the inverse of sddscollapse.

• sddsregroup -- Swaps the row indexing and page indexing of data in an SDDS file. That is, the ${\rm i {th}}$ row of the ${\rm j {th}}$ data page in the input file becomes the ${\rm j {th}}$ row of the ${\rm i {th}}$ data page of the output file. Example application: viewing the long-term evolution of a repeatedly-sampled waveform at each point in the waveform.

• sddstranspose -- Transposes the tabular data in the input file, so that the output file contains one column for each row in the input. Example usage: tranpose an orbit response matrix as part of preparing to use it for feedback.

• sddsmakedataset -- writes the input data into a file or pipe in SDDS format. It can be used to make add SDDS file consisting of a small amount of data from the script. It is more convenient than sdds save.

• sddsmatrixmult -- Multiplies the tabular data in the two input files to produce a file containing a matrix of the product. Example usage: Multiply a vector of errors with a correction matrix to obtain a vector of corrections to apply in a step-by-step feedback system.

• sddsmatrixop -- performs general matrix operations. The matrices and operations are specified on the command line and the operations will proceed in a rpn-like fashion.

• sddsselect -- Copies rows from one file based on the presence or absence of matching data in another file. Example application: finding all of the rows from one file that do not appear in a second file.

• sddssort -- Sorts the tabular data section of a data set by the values in named columns. Optionally eliminates duplicate rows.
• sddssortcolumn -- rearrange the columns of an SDDS data.
• sddssplit -- Places each page of a file in a separate, new file. Example application: getting selected pages of a file into separate, single-page files for use with a program that only recognizes the first page.

• sddsxref -- Creates a new data set by adding selected rows from one data set to another data set. Example application: cross-referencing the turn-by-turn coordinates of particles in a tracking simulation with the initial coordinates using a particle ID number.