A Brief Introduction to the SDDS Toolkit

Michael Borland

Table of Contents

A Brief Introduction to the SDDS ToolkitThe SDDS Toolkit
Michael Borland
Operations Analysis Group
What do the following items from recent operations logbooks have in common?

SCR files of the injectors were saved.
RMS beam motion is: x=4.028um, y=1.821um.
102mA stored beam, running orbit correction.
Storage Ring tunes are: x=0.1948, y=0.2753.
The Storage Ring RF configuration was compared to the reference file.
Steering complete for 33ID.
G. Decker is collecting fast beam history data.
No problems detected with 48 data loggers.
Louis Emery topped up the ring to 100mA using the top-up software.
Singlet bunch pattern studies were performed by M. Borland until R. Merl arrived to do the top-up current monitor studies.

All of these entries are referring to activities that depend on a software system called the SDDS Toolkit.

The Toolkit was not designed with any of these activities in mind.

Paradoxically, this is why it is so useful.

What is SDDS?

SDDS stands for "Self Describing Data Sets."
SDDS is just a standardized way to store and access data, i.e., a "file protocol."
SDDS also refers to a group of ~85 programs that use this file protocol.
These programs are the "tools" in the SDDS Toolkit.

The Toolkit Analogy

A hammer, a saw, a drill, etc., can be used together or sequentially to create and modify physical objects.
The programs in the SDDS toolkit can be used to sequentially transform SDDS data sets.

Within some limits, it isn't determined ahead of time what physical objects can be modified or what can be created.
SDDS toolkit programs are generic and operate on any SDDS data set. The meaning of the operations is not predetermined.
Both physical and SDDS tools can be used in arbitrary sequences of arbitrary length. The capability of the toolkit grows very rapidly with the number of tools.

Every new tool that is created makes the existing ones more useful, without any advance planning or coordination by developers.

A new tool need not be useful by itself in order to be very useful as part of a toolkit.
Most SDDS tools produce no directly useful result. This freedom makes new tools much easier to create.

Conventional Paradigm

SDDS-Compliant Programs

Three types of SDDS-compliant programs

SDDS Toolkit Paradigm

Examples of SDDS Tool Functions

Data display
plotting (2 programs)
printing data as formatted text
summarizing data set contents
Data processing
equation evaluation
data filtering and outlier removal
statistics, histograms, and correlations
fitting and smoothing
matrix operations (e.g., SVD)
cross-referencing, sorting, and collation
FFTs and digital filtering
Data collection from EPICS
logging data at fixed time intervals
event-driven data logging
alarm logging
n-dimensional experiments
save/restore of EPICS data
Control functions for EPICS
generalized feedback control
generalized optimization

Making Tools into Applications

Because the SDDS tools are commandline driven, they can be embedded in scripts..
Tcl/Tk is used to make graphical user interfaces (GUIs) that depend on SDDS tools for computational "muscle," data collection, and data display.
Engineers and physicists can use SDDS tools directly to develop new algorithms. Once finished, those results can easily be put into a GUI script.

How the SDDS Toolkit is Used at APS

Automated data collection
~19000 channels of time-series data
~1000 channels of glitch data
~9000 channels of alarm data
storage ring beam dump data
Used to create high-level applications for operators, engineers, and physicists:
beamline steering
orbit correction
configuration control
routine monitoring
history review
problem diagnosis
Used by engineers and physicists for
data collection
automated experiments
analysis of accelerator and simulation data
equipment checkout

Self Describing Data Files

Self-describing data files require more information in the file besides the data itself.
At minimum, a self-describing file protocol
requires that every data element in the file has a name.
forbids access to data except via the name.

The SDDS File Protocol

SDDS is a specific self-describing data protocol, developed at APS for accelerator commissioning.
Highly successful application of SDDS to commissioning lead to its use for operations.
An SDDS file consists of
A file header describing a structure composed of an arbitrary number of parameters and arrays, and a data table of arbitrary rows and columns.
Zero or more instances of the structure.
There are many extremely general self-describing file protocols around today.
In using these protocols, users find it necessary to create elaborate data standards of their own, which inhibits use of the toolkit approach.
In contrast, the SDDS file protocol is simple enough to be used in "daily life," but general enough to be widely useful.
Only the simplicity of the data model makes the SDDS Toolkit feasible.

Examples of SDDS Files

Accelerator configuration data:
Parameters: time stamp, configuration description, username, etc.
Columns: process variable name, value, access mode, category, subcategory, tolerance, etc.
Storage ring orbit glitch records
Parameters: time of glitch, trigger conditions, etc.
Columns: readouts of all BPMs, time of readout, beam current, etc.

Interest in SDDS Outside APS

IPNS---Used for data logging, analysis, and display.
CEA (France)---Used by a group of particle physicists doing underwater experiments in the Mediteranean.
SRRC (Taiwan)---Installed by request.
LEDA (LANL)---Installed by request.
CEBAF (TJNAF)---Installed by request.

To Learn More

Look for an upcoming four-part class on using SDDS.
See the OAG web page at
APS Overview --->
Accelerator Systems Division --->
Operations Analysis --->
OAG Software Documentation
In particular, see

SDDS tools, and
SDDS Compliant EPICS tools.