A Brief Introduction to the SDDS Toolkit

The SDDS Toolkit • Michael Borland • Accelerator Operations Physics

What do the following items from recent operations logbooks have in common?

SCR files of the injectors were saved.
RMS beam motion is: x = 4.028 μm, y = 1.821 μm.
102 mA stored beam, running orbit correction.
Storage Ring tunes are: x = 0.1948, y = 0.2753.
The Storage Ring RF configuration was compared to the reference file.
Steering complete for 33ID.
G. Decker is collecting fast beam history data.
No problems detected with 48 data loggers.
Louis Emery topped up the ring to 100 mA using the top-up software.
Singlet bunch pattern studies were performed by M. Borland until R. Merl arrived to do the top-up current monitor studies.

Answer: All of these entries are referring to activities that depend on a software system called the SDDS Toolkit.

The Toolkit was not designed with any of these activities in mind. Paradoxically, this is why it is so useful.

What is SDDS?

A hammer, a saw, a drill, etc., can be used together or sequentially to create and modify physical objects.
The programs in the SDDS Toolkit can be used to sequentially transform SDDS data sets.
Within some limits, it isn’t determined ahead of time what physical objects can be modified or what can be created.
SDDS Toolkit programs are generic and operate on any SDDS data set. The meaning of the operations is not predetermined.
Both physical and SDDS tools can be used in arbitrary sequences of arbitrary length. The capability of the toolkit grows very rapidly with the number of tools.
Every new tool that is created makes the existing ones more useful, without any advance planning or coordination by developers.
A new tool need not be useful by itself to be very useful as part of a toolkit. Most SDDS tools produce no directly useful result. This freedom makes new tools much easier to create.

Three types of SDDS-compliant programs:

Data display
- plotting (2 programs)
- printing data as formatted text
- summarizing data set contents
Data processing
- equation evaluation
- data filtering and outlier removal
- statistics, histograms, and correlations
- fitting and smoothing
- matrix operations (e.g., SVD)
- cross-referencing, sorting, and collation
- FFTs and digital filtering
Data collection from EPICS
- logging data at fixed time intervals
- event-driven data logging
- alarm logging
- n-dimensional experiments
- save/restore of EPICS data
Control functions for EPICS
- generalized feedback control
- generalized optimization

Because the SDDS tools are command-line driven, they can be embedded in scripts.
Tcl/Tk is used to make graphical user interfaces (GUIs) that depend on SDDS tools for computational “muscle,” data collection, and data display.
Engineers and physicists can use SDDS tools directly to develop new algorithms. Once finished, those results can easily be put into a GUI script.

Automated data collection
- ~19,000 channels of time-series data
- ~1,000 channels of glitch data
- ~9,000 channels of alarm data
- storage ring beam dump data
Used to create high-level applications for operators, engineers, and physicists:
- beamline steering
- orbit correction
- configuration control
- routine monitoring
- history review
- problem diagnosis
Used by engineers and physicists for
- data collection
- automated experiments
- analysis of accelerator and simulation data
- equipment checkout

Self-describing data files require more information in the file besides the data itself.
At minimum, a self-describing file protocol
- requires that every data element in the file has a name.
- forbids access to data except via the name.

SDDS is a specific self-describing data protocol, developed at APS for accelerator commissioning.
Highly successful application of SDDS to commissioning led to its use for operations.
An SDDS file consists of
- a file header describing a structure composed of an arbitrary number of parameters and arrays, and a data table of arbitrary rows and columns, and
- zero or more instances of the structure.
There are many extremely general self-describing file protocols around today.
In using these protocols, users find it necessary to create elaborate data standards of their own, which inhibits use of the toolkit approach.
In contrast, the SDDS file protocol is simple enough to be used in “daily life,” but general enough to be widely useful.
Only the simplicity of the data model makes the SDDS Toolkit feasible.

Accelerator configuration data:
- Parameters: time stamp, configuration description, username, etc.
- Columns: process variable name, value, access mode, category, subcategory, tolerance, etc.
Storage ring orbit glitch records
- Parameters: time of glitch, trigger conditions, etc.
- Columns: readouts of all BPMs, time of readout, beam current, etc.

IPNS - Used for data logging, analysis, and display.
CEA (France) - Used by a group of particle physicists doing underwater experiments in the Mediterranean.
SRRC (Taiwan) - Installed by request.
LEDA (LANL) - Installed by request.
CEBAF (TJNAF) - Installed by request.

Look for an upcoming four-part class on using SDDS.
See the AOP web page:
- AOP Software Documentation
- In particular, see:
  - SDDS ToolKit
  - SDDS EPICS ToolKit