SDDS ToolKit Programs and Libraries for C and Python
•All Classes Files Functions Variables Macros Pages
sddsprocess.c
Go to the documentation of this file.
1/**
2 * @file sddsprocess.c
3 * @brief Processes SDDS files by applying various transformations to columns and parameters.
4 *
5 * This program reads an input SDDS (Self Describing Data Sets) file, processes its columns and parameters
6 * based on the specified options, and writes the results to a new SDDS output file. It supports a wide
7 * range of operations including filtering, matching, editing, defining new parameters or columns,
8 * converting units, and more.
9 *
10 * @section Usage
11 * ```
12 * sddsprocess [<SDDSinputfile>] [<SDDSoutputfile>]
13 * -pipe=[input][,output]
14 * -ifis={column|parameter},<name>[,...]
15 * -ifnot={column|parameter},<name>[,...]
16 * -match={column|parameter},<match-test>[,<match-test>[,<logic-operation>...]]
17 * -filter={column|parameter},<range-spec>[,<range-spec>[,<logic-operation>...]]
18 * -timeFilter={parameter|column},[,before=YYYY/MM/DD@HH:MM:SS][,after=YYYY/MM/DD@HH:MM:SS][,invert]
19 * -clip=<head>,<tail>,[invert]
20 * -sparse=<interval>[,<offset>]
21 * -sample=<fraction>
22 * -fclip=<head>,<tail>,[invert]
23 * -test={column|parameter},<test>[,autostop][,algebraic]
24 * -numberTest={column|parameter},<name>[,invert]
25 * -rpndefinitionsfiles=<filename>[,...]
26 * -rpnexpression=<expression>[,repeat][,algebraic]
27 * -convertunits={column|parameter},<name>,<new-units>,<old-units>[,<factor>]
28 * -define={column|parameter},<name>,{<equation>|@<parameterName>}[,<definition_entries>][,algebraic]
29 * -redefine={column|parameter},<name>,{<equation>|@<parameterName>}[,<definition_entries>][,algebraic]
30 * -cast={column|parameter},<newName>,<sourceName>,<newType>
31 * -scan={column|parameter},<new-name>,<source-name>,<sscanf-string>[,<definition-entries>][,edit=<string>]
32 * -edit={column|parameter},<new-name>,<source-name>,<edit-string>[,<definition-entries>]
33 * -reedit={column|parameter},<name>,<edit-string>[,<definition-entries>]
34 * -print={column|parameter},<new-name>,<sprintf-string>,<source-name>[,...][,<definition-entries>]
35 * -reprint={column|parameter},<new-name>,<sprintf-string>,<source-name>[,...][,<definition-entries>]
36 * -format={column|parameter},<new-name>,<source-name>[,stringFormat=<sprintfString>][,doubleFormat=<sprintfString>][,longFormat=<sprintfString>]
37 * -system={column|parameter},<new-name>,<command-name>[,<definition_entries>]
38 * -evaluate={column|parameter},<new-name>,<expression-name>[,<definition_entries>]
39 * -process=<column-name>,<analysis-name>,<result-name>[,description=<string>][,symbol=<string>][,weightBy=<column-name>][,functionOf=<column-name>[,lowerLimit=<value>][,upperLimit=<value>][,position]][,head=<number>][,tail=<number>][fhead=<fraction>][ftail=<fraction>][,topLimit=<value>][,bottomLimit=<value>][,offset=<value>][,factor=<value>][match=<column-name>,value=<string>][,overwrite],[default=<value>]
40 * -summarize
41 * -verbose
42 * -nowarnings
43 * -delete={column|parameter|array},<matching-string>[,...]
44 * -retain={column|parameter|array},<matching-string>[,...]
45 * -description=[text=<string>][,contents=<string>]
46 * -majorOrder=row|column
47 * -threads=<number>
48 * ```
49 *
50 * @section Options
51 * | Optional | Description |
52 * |-------------------------------------|----------------------------------------------------------------------------------|
53 * | `-pipe` | Directs input/output via pipes. |
54 * | `-ifis` | Include only the specified columns or parameters. |
55 * | `-ifnot` | Exclude the specified columns or parameters. |
56 * | `-match` | Filter data based on matching criteria for columns or parameters. |
57 * | `-filter` | Apply range-based filtering on specified columns or parameters. |
58 * | `-timeFilter` | Filter data based on time constraints. |
59 * | `-clip` | Clip the dataset by removing a specified number of rows from the head and tail. |
60 * | `-sparse` | Retain every `<interval>`-th row starting from `<offset>`. |
61 * | `-sample` | Randomly sample rows based on the specified fraction. |
62 * | `-fclip` | Fractional clipping based on the fraction of rows. |
63 * | `-test` | Apply RPN-based tests on columns or parameters with optional autostop. |
64 * | `-numberTest` | Perform numerical tests on specified columns or parameters. |
65 * | `-rpndefinitionsfiles` | Specify files containing RPN definitions. |
66 * | `-rpnexpression` | Define RPN expressions to be evaluated. |
67 * | `-convertunits` | Convert units of a specified column or parameter. |
68 * | `-define` | Define new columns or parameters based on equations or existing parameters. |
69 * | `-redefine` | Redefine existing columns or parameters. |
70 * | `-cast` | Cast a column or parameter to a new data type. |
71 * | `-scan` | Scan and parse data from a source column or parameter. |
72 * | `-edit` | Edit values of a column or parameter based on specified rules. |
73 * | `-reedit` | Re-edit existing columns or parameters. |
74 * | `-print` | Print formatted data from a source to a new column or parameter. |
75 * | `-reprint` | Reprint existing columns or parameters with new formatting. |
76 * | `-format` | Apply formatting to columns or parameters. |
77 * | `-system` | Execute system commands and capture their output. |
78 * | `-evaluate` | Evaluate expressions to compute new columns or parameters. |
79 * | `-process` | Perform various processing operations on specified columns. |
80 * | `-summarize` | Display a summary of the processing operations performed. |
81 * | `-verbose` | Enable verbose output for detailed processing information. |
82 * | `-nowarnings` | Suppress warning messages. |
83 * | `-delete` | Delete specified columns, parameters, or arrays. |
84 * | `-retain` | Retain only the specified columns, parameters, or arrays. |
85 * | `-description` | Add or modify the description of the output SDDS file. |
86 * | `-majorOrder` | Set the major order of the output data (row-major or column-major). |
87 * | `-threads` | Specify the number of threads to use for processing. |
88 *
89 * @section processingModes Processing Modes
90 * | Modes | | | |
91 * |----------------|---------------|--------------|---------------------|
92 * | average | rms | sum | standarddeviation |
93 * | mad | minimum | maximum | smallest |
94 * | largest | first | last | count |
95 * | spread | median | baselevel | toplevel |
96 * | amplitude | risetime | falltime | fwhm |
97 * | fwtm | center | zerocrossing | fwha |
98 * | fwta | sigma | slope | intercept |
99 * | lfsd | qrange | drange | percentile |
100 * | mode | integral | product | prange |
101 * | signedsmallest | signedlargest | gmintegral | correlation |
102 *
103 * @subsection additonalInformation Additional Information
104 * - `<match-test>` is of the form `<name>=<matching-string>[,!]`, where ! signifies logical negation.
105 * - `<logic-operation>` is one of & (logical and) or | (logical or), optionally followed by a ! to logically negate the value of the expression.
106 * - `<range-spec>` is of the form `<name>,<lower-value>,<upper-value>[,!]`.
107 * - `<test>` is an RPN expression leaving a value on the logic stack.
108 * - `<definition-entry>` is of the form `<entry-name>=<value>`, where `<entry-name>` is one of "symbol", "units", "description", "format_string", and "type".
109 *
110 * @copyright
111 * - (c) 2002 The University of Chicago, as Operator of Argonne National Laboratory.
112 * - (c) 2002 The Regents of the University of California, as Operator of Los Alamos National Laboratory.
113 *
114 * @license
115 * This file is distributed under the terms of the Software License Agreement
116 * found in the file LICENSE included with this distribution.
117 *
118 * @author M. Borland, C. Saunders, R. Soliday, H. Shang
119 */
120
121#include "mdb.h"
122#include "scan.h"
123#include "SDDS.h"
124#include "SDDSaps.h"
125#include "rpn.h"
126#include <sys/types.h>
127#if !defined(_WIN32)
128# include <sys/time.h>
129#endif
130
131#define COLUMN_MODE 0
132#define PARAMETER_MODE 1
133#define ARRAY_MODE 2
134#define MODES 3
135 static char *mode_name[MODES] = {
136 "column",
137 "parameter",
138 "array",
139 };
140
141/* Enumeration for option types */
142enum option_type {
143 SET_PROCESS,
144 SET_MATCH,
145 SET_FILTER,
146 SET_RPN_DEFNS_FILES,
147 SET_RPN_EXPRESSION,
148 SET_DEFINITION,
149 SET_TEST,
150 SET_SUMMARIZE,
151 SET_VERBOSE,
152 SET_CONVERT_UNITS,
153 SET_SCAN,
154 SET_PRINT,
155 SET_EDIT,
156 SET_SYSTEM,
157 SET_REDEFINITION,
158 SET_NOWARNINGS,
159 SET_IFNOT,
160 SET_IFIS,
161 SET_CLIP,
162 SET_SPARSE,
163 SET_SAMPLE,
164 SET_PIPE,
165 SET_NUMBERTEST,
166 SET_REEDIT,
167 SET_FORMAT,
168 SET_CAST,
169 SET_REPRINT,
170 SET_DELETE,
171 SET_RETAIN,
172 SET_DESCRIPTION,
173 SET_FCLIP,
174 SET_TIME_FILTER,
175 SET_MAJOR_ORDER,
176 SET_EVALUATE,
177 SET_THREADS,
178 N_OPTIONS
179};
180
181char *option[N_OPTIONS] = {
182 "process",
183 "match",
184 "filter",
185 "rpndefinitionsfiles",
186 "rpnexpression",
187 "define",
188 "test",
189 "summarize",
190 "verbose",
191 "convertunits",
192 "scan",
193 "print",
194 "edit",
195 "system",
196 "redefine",
197 "nowarnings",
198 "ifnot",
199 "ifis",
200 "clip",
201 "sparse",
202 "sample",
203 "pipe",
204 "numbertest",
205 "reedit",
206 "format",
207 "cast",
208 "reprint",
209 "delete",
210 "retain",
211 "description",
212 "fclip",
213 "timeFilter",
214 "majorOrder",
215 "evaluate",
216 "threads",
217};
218
219char *usageArray[] = {
220 "Usage:\n",
221 " sddsprocess [<SDDSinputfile>] [<SDDSoutputfile>] [OPTIONS]\n",
222 "\n",
223 "Options:\n",
224 " -pipe=[input][,output]\n",
225 " Use pipe mode with optional input and output streams.\n",
226 " -ifis={column|parameter},<name>[,...]\n",
227 " Include only the specified columns or parameters.\n",
228 " -ifnot={column|parameter},<name>[,...]\n",
229 " Exclude the specified columns or parameters.\n",
230 " -match={column|parameter},<match-test>[,<match-test>,...]\n",
231 " Filter data based on matching criteria for columns or parameters.\n",
232 " -filter={column|parameter},<range-spec>[,<range-spec>,...]\n",
233 " Apply range-based filtering on specified columns or parameters.\n",
234 " -timeFilter={parameter|column},[,before=YYYY/MM/DD@HH:MM:SS][,after=YYYY/MM/DD@HH:MM:SS][,invert]\n",
235 " Filter data based on time constraints.\n",
236 " -clip=<head>,<tail>,[invert]\n",
237 " Clip the dataset by removing a specified number of rows from the head and tail.\n",
238 " -sparse=<interval>[,<offset>]\n",
239 " Retain every <interval>-th row starting from <offset>.\n",
240 " -sample=<fraction>\n",
241 " Randomly sample rows based on the specified fraction.\n",
242 " -fclip=<head>,<tail>,[invert]\n",
243 " Fractional clipping based on the fraction of rows.\n",
244 " -test={column|parameter},<test>[,autostop][,algebraic]\n",
245 " Apply RPN-based tests on columns or parameters with optional autostop.\n",
246 " -numberTest={column|parameter},<name>[,invert]\n",
247 " Perform numerical tests on specified columns or parameters.\n",
248 " -rpndefinitionsfiles=<filename>[,...]\n",
249 " Specify files containing RPN definitions.\n",
250 " -rpnexpression=<expression>[,repeat][,algebraic]\n",
251 " Define RPN expressions to be evaluated.\n",
252 " -convertunits={column|parameter},<name>,<new-units>,<old-units>[,<factor>]\n",
253 " Convert units of a specified column or parameter.\n",
254 " -define={column|parameter},<name>,{<equation>|@<parameterName>}[,<definition_entries>][,algebraic]\n",
255 " Define new columns or parameters based on equations or existing parameters.\n",
256 " -redefine={column|parameter},<name>,{<equation>|@<parameterName>}[,<definition_entries>][,algebraic]\n",
257 " Redefine existing columns or parameters.\n",
258 " -cast={column|parameter},<newName>,<sourceName>,<newType>\n",
259 " Cast a column or parameter to a new data type.\n",
260 " -scan={column|parameter},<new-name>,<source-name>,<sscanf-string>[,<definition-entries>][,edit=<string>]\n",
261 " Scan and parse data from a source column or parameter.\n",
262 " -edit={column|parameter},<new-name>,<source-name>,<edit-string>[,<definition-entries>]\n",
263 " Edit values of a column or parameter based on specified rules.\n",
264 " -reedit={column|parameter},<name>,<edit-string>[,<definition-entries>]\n",
265 " Re-edit existing columns or parameters.\n",
266 " -print={column|parameter},<new-name>,<sprintf-string>,<source-name>[,...][,<definition-entries>]\n",
267 " Print formatted data from a source to a new column or parameter.\n",
268 " -reprint={column|parameter},<new-name>,<sprintf-string>,<source-name>[,...][,<definition-entries>]\n",
269 " Reprint existing columns or parameters with new formatting.\n",
270 " -format={column|parameter},<new-name>,<source-name>[,stringFormat=<sprintfString>][,doubleFormat=<sprintfString>][,longFormat=<sprintfString>]\n",
271 " Apply formatting to columns or parameters.\n",
272 " -system={column|parameter},<new-name>,<command-name>[,<definition_entries>]\n",
273 " Execute system commands and capture their output.\n",
274 " -evaluate={column|parameter},<new-name>,<expression-name>[,<definition_entries>]\n",
275 " Evaluate expressions to compute new columns or parameters.\n",
276 " -process=<column-name>,<analysis-name>,<result-name>[,description=<string>][,symbol=<string>][,weightBy=<column-name>][,functionOf=<column-name>[,lowerLimit=<value>][,upperLimit=<value>][,position]][,head=<number>][,tail=<number>][fhead=<fraction>][ftail=<fraction>][,topLimit=<value>][,bottomLimit=<value>][,offset=<value>][,factor=<value>][match=<column-name>,value=<string>][,overwrite],[default=<value>]\n",
277 " Perform various processing operations on specified columns.\n",
278 " -summarize\n",
279 " Display a summary of the processing operations performed.\n",
280 " -verbose\n",
281 " Enable verbose output for detailed processing information.\n",
282 " -nowarnings\n",
283 " Suppress warning messages.\n",
284 " -delete={column|parameter|array},<matching-string>[,...]\n",
285 " Delete specified columns, parameters, or arrays.\n",
286 " -retain={column|parameter|array},<matching-string>[,...]\n",
287 " Retain only the specified columns, parameters, or arrays.\n",
288 " -description=[text=<string>][,contents=<string>]\n",
289 " Add or modify the description of the output SDDS file.\n",
290 " -majorOrder=row|column\n",
291 " Set the major order of the output data (row-major or column-major).\n",
292 " -threads=<number>\n",
293 " Specify the number of threads to use for processing.\n",
294 "\n",
295 "Additional Information:\n",
296 " A <match-test> is of the form <name>=<matching-string>[,!], where ! signifies logical negation.\n",
297 " A <logic-operation> is one of & (logical and) or | (logical or), optionally followed by a ! to logically negate the value of the expression.\n",
298 " A <range-spec> is of the form <name>,<lower-value>,<upper-value>[,!].\n",
299 " A <test> is an RPN expression leaving a value on the logic stack.\n",
300 " A <definition-entry> is of the form <entry-name>=<value>, where <entry-name> is one of \"symbol\", \"units\", \"description\", \"format_string\", and \"type\".\n",
301 "\n",
302 "Description:\n",
303 " sddsprocess reads data from an SDDS file, processes it according to the specified options, and writes the results to a new SDDS file.\n",
304 "Program by Michael Borland. (" __DATE__ " " __TIME__ ", SVN revision: " SVN_VERSION ")\n",
305 NULL};
306
307#define PARAMETER_OR_COLUMN(is_param) ((is_param) ? "parameter" : "column")
308
309#define REDEFINABLE_TYPE(type) (SDDS_NUMERIC_TYPE(type) || (type) == SDDS_CHARACTER)
310
311/*
312 static char *colpar_choice[2] = {"column", "parameter"};
313*/
314static long table_number_mem = -1, i_page_mem = -1, n_rows_mem = -1, i_row_mem = -1;
315
316typedef char *STRING_PAIR[2];
317
318typedef struct
319{
320 char *match_string, *edit_string;
321} EDIT_NAME_REQUEST;
322
323char **process_name_options(char **orig_name, long **orig_flag, long orig_names, char **delete, long deletes,
324 char **retain, long retains, STRING_PAIR *rename, long renames,
325 EDIT_NAME_REQUEST *edit_request, long edit_requests);
326long SDDS_ComputeSetOfColumns(SDDS_DATASET *SDDS_dataset, long equ_begin, long equ_end, DEFINITION *definition);
327void free_processing_definiton(PROCESSING_DEFINITION **processing_definition, long n);
328void free_equation_definition(EQUATION_DEFINITION **equation_definition, long n);
329long SDDS_EvaluateColumn(SDDS_DATASET *SDDS_dataset, EVALUATE_DEFINITION *definition);
330long SDDS_EvaluateParameter(SDDS_DATASET *SDDS_dataset, EVALUATE_DEFINITION *definition);
331
332int main(int argc, char **argv) {
333 SDDS_DATASET SDDS_input, SDDS_output;
334 PARAMETER_DEFINITION *pardefptr;
335 //long output_columns;
336 char *description_text, *description_contents;
337 unsigned long flags;
338
339 PROCESSING_DEFINITION **processing_definition, *processing_ptr;
340 long processing_definitions;
341 CONVERSION_DEFINITION **conversion, *conversion_ptr;
342 long conversions;
343 EDIT_DEFINITION **edit_definition, *edit_ptr;
344 long edit_definitions;
345 EQUATION_DEFINITION **equation_definition, *equation_ptr;
346 long equation_definitions;
347 EVALUATE_DEFINITION **evaluate_definition, *evaluate_ptr;
348 long evaluate_definitions;
349 SCAN_DEFINITION **scan_definition, *scan_ptr;
350 long scan_definitions;
351 PRINT_DEFINITION **print_definition, *print_ptr;
352 long print_definitions;
353 FORMAT_DEFINITION **format_definition, *format_ptr;
354 long format_definitions;
355 FILTER_TERM *filter_term;
356 FILTER_DEFINITION **filter_definition, *filter_ptr;
357 long filter_definitions;
358 TIME_FILTER_DEFINITION **time_filter_definition, *time_filter_ptr;
359 long time_filter_definitions;
360 MATCH_TERM *match_term;
361 MATCH_DEFINITION **match_definition, *match_ptr;
362 long match_definitions;
363 RPNTEST_DEFINITION **rpntest_definition, *rpntest_ptr;
364 long rpntest_definitions;
365 NUMBERTEST_DEFINITION **numbertest_definition, *numbertest_ptr;
366 long numbertest_definitions;
367 SYSTEM_DEFINITION **system_definition, *system_ptr;
368 long system_definitions;
369 RPNEXPRESSION_DEFINITION **rpnexpression_definition, *rpnexpression_ptr;
370 long rpnexpression_definitions;
371 CLIP_DEFINITION **clip_definition, *clip_ptr;
372 FCLIP_DEFINITION **fclip_definition, *fclip_ptr;
373 long clip_definitions, fclip_definitions;
374 SPARSE_DEFINITION **sparse_definition, *sparse_ptr;
375 long sparse_definitions;
376 SAMPLE_DEFINITION **sample_definition, *sample_ptr;
377 long sample_definitions;
378 DEFINITION *definition;
379 long definitions;
380 CAST_DEFINITION **cast_definition, *cast_ptr;
381 long cast_definitions;
382
383 IFITEM_LIST ifnot_item, ifis_item;
384 char *inputfile, *outputfile;
385
386 int64_t n_rows, j;
387 long i, k, i_arg, summarize, verbose, index = 0, tmpfile_used, nowarnings, udfCounter;
388 long page_number, accept;
389 SCANNED_ARG *s_arg;
390 char s[SDDS_MAXLINE];
391
392 char **rpn_definitions_file, *stringResult;
393 long rpn_definitions_files;
394 int32_t test_result;
395 long original_parameters;
396 double result;
397 long i_page_memory, table_number_memory, n_rows_memory, row_deletion, skip_page;
398 long random_number_seed;
399 unsigned long pipeFlags, majorOrderFlag;
400
401 char **retain_column, **delete_column;
402 long retain_columns, delete_columns, rename_columns, edit_column_requests;
403 EDIT_NAME_REQUEST *edit_column_request;
404 STRING_PAIR *rename_column;
405 char **orig_column_name, **new_column_name;
406 long *orig_column_flag;
407 int32_t orig_column_names;
408
409 char **retain_parameter, **delete_parameter;
410 long retain_parameters, delete_parameters, rename_parameters, edit_parameter_requests;
411 EDIT_NAME_REQUEST *edit_parameter_request;
412 STRING_PAIR *rename_parameter;
413 char **orig_parameter_name, **new_parameter_name;
414 long *orig_parameter_flag;
415 int32_t orig_parameter_names;
416
417 char **retain_array, **delete_array;
418 long retain_arrays, delete_arrays, rename_arrays, edit_array_requests;
419 EDIT_NAME_REQUEST *edit_array_request;
420 STRING_PAIR *rename_array;
421 char **orig_array_name, **new_array_name;
422 long *orig_array_flag;
423 int32_t orig_array_names;
424 short columnMajorOrder = -1;
425
426 int threads = 1;
427
428 char buffer[SDDS_MAXLINE];
429
430 SDDS_RegisterProgramName(argv[0]);
431
432 argc = scanargs(&s_arg, argc, argv);
433 argc = add_sddsfile_arguments(&s_arg, argc);
434
435 random_number_seed = (long)time((time_t *)NULL);
436 random_number_seed = 2 * (random_number_seed / 2) + 1;
437#if defined(_WIN32)
438 random_1(-labs((double)random_number_seed));
439#else
440 random_1(-FABS((double)random_number_seed));
441#endif
442
443 retain_column = delete_column = NULL;
444 retain_columns = delete_columns = rename_columns = edit_column_requests = 0;
445 rename_column = NULL;
446 edit_column_request = NULL;
447
448 retain_parameter = delete_parameter = NULL;
449 retain_parameters = delete_parameters = rename_parameters = edit_parameter_requests = 0;
450 rename_parameter = NULL;
451 edit_parameter_request = NULL;
452
453 retain_array = delete_array = NULL;
454 retain_arrays = delete_arrays = rename_arrays = edit_array_requests = 0;
455 rename_array = NULL;
456 edit_array_request = NULL;
457
458 description_text = description_contents = NULL;
459 new_parameter_name = new_column_name = new_array_name = NULL;
460
461 if (argc < 3) {
462 char **ptr;
463 ptr = usageArray;
464 while (*ptr) {
465 fprintf(stderr, "%s", *ptr);
466 ptr++;
467 }
468 show_process_modes(stderr);
469 exit(EXIT_FAILURE);
470 }
471
472 inputfile = outputfile = NULL;
473 rpn_definitions_file = NULL;
474 rpn_definitions_files = 0;
475 summarize = verbose = nowarnings = udfCounter = 0;
476
477 processing_definition = NULL;
478 processing_definitions = 0;
479 conversion = NULL;
480 conversions = 0;
481 edit_definition = NULL;
482 edit_definitions = 0;
483 equation_definition = NULL;
484 equation_definitions = 0;
485 evaluate_definition = NULL;
486 evaluate_definitions = 0;
487 scan_definition = NULL;
488 scan_definitions = 0;
489 print_definition = NULL;
490 print_definitions = 0;
491 format_definition = NULL;
492 format_definitions = 0;
493 filter_definition = NULL;
494 filter_definitions = 0;
495 time_filter_definition = NULL;
496 time_filter_definitions = 0;
497 match_definition = NULL;
498 match_definitions = 0;
499 rpntest_definition = NULL;
500 rpntest_definitions = 0;
501 numbertest_definition = NULL;
502 numbertest_definitions = 0;
503 system_definition = NULL;
504 system_definitions = 0;
505 rpnexpression_definition = NULL;
506 rpnexpression_definitions = 0;
507 clip_definition = NULL;
508 fclip_definition = NULL;
509 clip_definitions = fclip_definitions = 0;
510 sparse_definition = NULL;
511 sparse_definitions = 0;
512 sample_definition = NULL;
513 sample_definitions = 0;
514 cast_definition = NULL;
515 cast_definitions = 0;
516 definition = NULL;
517 definitions = 0;
518 ifnot_item.items = 0;
519 ifis_item.items = 0;
520 pipeFlags = 0;
521
522 for (i_arg = 1; i_arg < argc; i_arg++) {
523 if (s_arg[i_arg].arg_type == OPTION) {
524 delete_chars(s_arg[i_arg].list[0], "_");
525 switch (match_string(s_arg[i_arg].list[0], option, N_OPTIONS, 0)) {
526 case SET_MAJOR_ORDER:
527 majorOrderFlag = 0;
528 s_arg[i_arg].n_items--;
529 if (s_arg[i_arg].n_items > 0 &&
530 (!scanItemList(&majorOrderFlag, s_arg[i_arg].list + 1, &s_arg[i_arg].n_items, 0,
531 "row", -1, NULL, 0, SDDS_ROW_MAJOR_ORDER,
532 "column", -1, NULL, 0, SDDS_COLUMN_MAJOR_ORDER, NULL)))
533 SDDS_Bomb("invalid -majorOrder syntax/values");
534 if (majorOrderFlag & SDDS_COLUMN_MAJOR_ORDER)
535 columnMajorOrder = 1;
536 else if (majorOrderFlag & SDDS_ROW_MAJOR_ORDER)
537 columnMajorOrder = 0;
538 break;
539 case SET_MATCH:
540 if (s_arg[i_arg].n_items < 3)
541 SDDS_Bomb("invalid -match syntax");
542 match_definition = (MATCH_DEFINITION **)trealloc(match_definition, sizeof(*match_definition) * (match_definitions + 1));
543 if (!(match_definition[match_definitions] = process_new_match_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
544 SDDS_Bomb("invalid -match syntax");
545 add_definition(&definition, &definitions, (void *)match_definition[match_definitions], IS_MATCH_DEFINITION);
546 match_definitions++;
547 break;
548 case SET_FILTER:
549 if (s_arg[i_arg].n_items < 3)
550 SDDS_Bomb("invalid -filter syntax");
551 filter_definition = (FILTER_DEFINITION **)trealloc(filter_definition, sizeof(*filter_definition) * (filter_definitions + 1));
552 if (!(filter_definition[filter_definitions] = process_new_filter_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
553 SDDS_Bomb("invalid -filter syntax");
554 add_definition(&definition, &definitions, (void *)filter_definition[filter_definitions], IS_FILTER_DEFINITION);
555 filter_definitions++;
556 break;
557 case SET_TIME_FILTER:
558 if (s_arg[i_arg].n_items < 3)
559 SDDS_Bomb("invalid -timeFilter syntax");
560 time_filter_definition = (TIME_FILTER_DEFINITION **)trealloc(time_filter_definition, sizeof(*time_filter_definition) * (filter_definitions + 1));
561 if (!(time_filter_definition[time_filter_definitions] = process_new_time_filter_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
562 SDDS_Bomb("invalid -timeFilter syntax");
563 add_definition(&definition, &definitions, (void *)time_filter_definition[time_filter_definitions], IS_TIME_FILTER_DEFINITION);
564 time_filter_definitions++;
565 break;
566 case SET_RPN_DEFNS_FILES:
567 if (s_arg[i_arg].n_items < 2)
568 SDDS_Bomb("invalid -rpndefinitionsfiles syntax");
569 rpn_definitions_file = trealloc(rpn_definitions_file, sizeof(*rpn_definitions_file) * (rpn_definitions_files + s_arg[i_arg].n_items));
570 for (i = 1; i < s_arg[i_arg].n_items; i++)
571 if (!fexists(rpn_definitions_file[rpn_definitions_files + i - 1] = s_arg[i_arg].list[i]))
572 SDDS_Bomb("one or more rpn definitions files do not exist");
573 rpn_definitions_files += s_arg[i_arg].n_items - 1;
574 break;
575 case SET_RPN_EXPRESSION:
576 if (s_arg[i_arg].n_items < 2 && s_arg[i_arg].n_items > 4)
577 SDDS_Bomb("invalid -rpnexpression syntax");
578 rpnexpression_definition = (RPNEXPRESSION_DEFINITION **)trealloc(rpnexpression_definition, sizeof(*rpnexpression_definition) * (rpnexpression_definitions + 1));
579 if (!(rpnexpression_definition[rpnexpression_definitions] = process_new_rpnexpression_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
580 SDDS_Bomb("invalid -rpnexpression syntax");
581 add_definition(&definition, &definitions, (void *)rpnexpression_definition[rpnexpression_definitions], IS_RPNEXPRESSION_DEFINITION);
582 rpnexpression_definitions++;
583 break;
584 case SET_DEFINITION:
585 if (s_arg[i_arg].n_items < 4)
586 SDDS_Bomb("invalid -definition syntax");
587 equation_definition = (EQUATION_DEFINITION **)trealloc(equation_definition, sizeof(*equation_definition) * (equation_definitions + 1));
588 if (!(equation_definition[equation_definitions] = process_new_equation_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
589 SDDS_Bomb("invalid -definition syntax");
590 equation_definition[equation_definitions]->redefinition = 0;
591 add_definition(&definition, &definitions, (void *)equation_definition[equation_definitions], IS_EQUATION_DEFINITION);
592 equation_definitions++;
593 break;
594 case SET_REDEFINITION:
595 if (s_arg[i_arg].n_items < 4)
596 SDDS_Bomb("invalid -redefinition syntax");
597 equation_definition = (EQUATION_DEFINITION **)trealloc(equation_definition, sizeof(*equation_definition) * (equation_definitions + 1));
598 if (!(equation_definition[equation_definitions] = process_new_equation_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
599 SDDS_Bomb("invalid -redefinition syntax");
600 equation_definition[equation_definitions]->redefinition = 1;
601 add_definition(&definition, &definitions, (void *)equation_definition[equation_definitions], IS_EQUATION_DEFINITION);
602 equation_definitions++;
603 break;
604 case SET_EVALUATE:
605 if (s_arg[i_arg].n_items < 4)
606 SDDS_Bomb("invalid -evaluate syntax");
607 evaluate_definition = (EVALUATE_DEFINITION **)trealloc(evaluate_definition, sizeof(*evaluate_definition) * (evaluate_definitions + 1));
608 if (!(evaluate_definition[evaluate_definitions] = process_new_evaluate_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
609 SDDS_Bomb("invalid -definition syntax");
610 add_definition(&definition, &definitions, (void *)evaluate_definition[evaluate_definitions], IS_EVALUATE_DEFINITION);
611 evaluate_definitions++;
612 break;
613 case SET_TEST:
614 if (s_arg[i_arg].n_items < 3 && s_arg[i_arg].n_items > 5)
615 SDDS_Bomb("invalid -test syntax");
616 rpntest_definition = (RPNTEST_DEFINITION **)trealloc(rpntest_definition, sizeof(*rpntest_definition) * (rpntest_definitions + 1));
617 if (!(rpntest_definition[rpntest_definitions] = process_new_rpntest_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
618 SDDS_Bomb("invalid -test syntax");
619 add_definition(&definition, &definitions, (void *)rpntest_definition[rpntest_definitions], IS_RPNTEST_DEFINITION);
620 rpntest_definitions++;
621 break;
622 case SET_SUMMARIZE:
623 summarize = 1;
624 break;
625 case SET_VERBOSE:
626 verbose = 1;
627 break;
628 case SET_NOWARNINGS:
629 nowarnings = 1;
630 break;
631 case SET_PROCESS:
632 processing_definition = (PROCESSING_DEFINITION **)trealloc(processing_definition, sizeof(*processing_definition) * (processing_definitions + 1));
633 if (!(processing_definition[processing_definitions] = record_processing_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
634 SDDS_Bomb("invalid -process syntax--parsing problem");
635 add_definition(&definition, &definitions, (void *)processing_definition[processing_definitions], IS_PROCESSING_DEFINITION);
636 processing_definitions++;
637 break;
638 case SET_CONVERT_UNITS:
639 if (s_arg[i_arg].n_items != 6)
640 SDDS_Bomb("invalid -convertunits syntax--wrong number of items");
641 conversion = (CONVERSION_DEFINITION **)trealloc(conversion, sizeof(*conversion) * (conversions + 1));
642 if (!(conversion[conversions] = process_conversion_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
643 SDDS_Bomb("invalid -convertunits syntax");
644 add_definition(&definition, &definitions, (void *)conversion[conversions], IS_CONVERSION_DEFINITION);
645 conversions++;
646 break;
647 case SET_NUMBERTEST:
648 /* -numberTest={col | param},<name>[,invert] */
649 if (s_arg[i_arg].n_items < 3)
650 SDDS_Bomb("invalid -numberTest syntax");
651 numbertest_definition = (NUMBERTEST_DEFINITION **)trealloc(numbertest_definition, sizeof(*numbertest_definition) * (numbertest_definitions + 1));
652 if (!(numbertest_definition[numbertest_definitions] = process_new_numbertest_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
653 SDDS_Bomb("invalid -numberTest syntax");
654 add_definition(&definition, &definitions, (void *)numbertest_definition[numbertest_definitions], IS_NUMBERTEST_DEFINITION);
655 numbertest_definitions++;
656 break;
657 case SET_SCAN:
658 if (s_arg[i_arg].n_items < 5)
659 SDDS_Bomb("invalid -scan syntax");
660 scan_definition = (SCAN_DEFINITION **)trealloc(scan_definition, sizeof(*scan_definition) * (scan_definitions + 1));
661 if (!(scan_definition[scan_definitions] = process_new_scan_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
662 SDDS_Bomb("invalid -scan syntax");
663 add_definition(&definition, &definitions, (void *)scan_definition[scan_definitions], IS_SCAN_DEFINITION);
664 scan_definitions++;
665 break;
666 case SET_PRINT:
667 if (s_arg[i_arg].n_items < 4)
668 SDDS_Bomb("invalid -print syntax");
669 print_definition = (PRINT_DEFINITION **)trealloc(print_definition, sizeof(*print_definition) * (print_definitions + 1));
670 if (!(print_definition[print_definitions] = process_new_print_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
671 SDDS_Bomb("invalid -print syntax");
672 print_definition[print_definitions]->reprint = 0;
673 add_definition(&definition, &definitions, (void *)print_definition[print_definitions], IS_PRINT_DEFINITION);
674 print_definitions++;
675 break;
676 case SET_REPRINT:
677 if (s_arg[i_arg].n_items < 4)
678 SDDS_Bomb("invalid -print syntax");
679 print_definition = (PRINT_DEFINITION **)trealloc(print_definition, sizeof(*print_definition) * (print_definitions + 1));
680 if (!(print_definition[print_definitions] = process_new_print_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
681 SDDS_Bomb("invalid -print syntax");
682 print_definition[print_definitions]->reprint = 1;
683 add_definition(&definition, &definitions, (void *)print_definition[print_definitions], IS_PRINT_DEFINITION);
684 print_definitions++;
685 break;
686 case SET_EDIT:
687 if (s_arg[i_arg].n_items < 5)
688 SDDS_Bomb("invalid -edit syntax");
689 edit_definition = (EDIT_DEFINITION **)trealloc(edit_definition, sizeof(*edit_definition) * (edit_definitions + 1));
690 if (!(edit_definition[edit_definitions] = process_new_edit_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1, 0)))
691 SDDS_Bomb("invalid -edit syntax");
692 add_definition(&definition, &definitions, (void *)edit_definition[edit_definitions], IS_EDIT_DEFINITION);
693 edit_definitions++;
694 break;
695 case SET_REEDIT:
696 if (s_arg[i_arg].n_items < 4)
697 SDDS_Bomb("invalid -reedit syntax");
698 edit_definition = (EDIT_DEFINITION **)trealloc(edit_definition, sizeof(*edit_definition) * (edit_definitions + 1));
699 if (!(edit_definition[edit_definitions] = process_new_edit_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1, 1)))
700 SDDS_Bomb("invalid -reedit syntax");
701 edit_definition[edit_definitions]->reedit = 1;
702 add_definition(&definition, &definitions, (void *)edit_definition[edit_definitions], IS_EDIT_DEFINITION);
703 edit_definitions++;
704 break;
705 case SET_SYSTEM:
706 if (s_arg[i_arg].n_items < 4)
707 SDDS_Bomb("invalid -system syntax");
708 system_definition = (SYSTEM_DEFINITION **)trealloc(system_definition, sizeof(*system_definition) * (system_definitions + 1));
709 if (!(system_definition[system_definitions] = process_new_system_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
710 SDDS_Bomb("invalid -system syntax");
711 add_definition(&definition, &definitions, (void *)system_definition[system_definitions], IS_SYSTEM_DEFINITION);
712 system_definitions++;
713 break;
714 case SET_IFNOT:
715 if (s_arg[i_arg].n_items < 3)
716 SDDS_Bomb("invalid -ifnot syntax");
717 add_ifitem(&ifnot_item, s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1);
718 break;
719 case SET_IFIS:
720 if (s_arg[i_arg].n_items < 3)
721 SDDS_Bomb("invalid -ifis syntax");
722 add_ifitem(&ifis_item, s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1);
723 break;
724 case SET_CLIP:
725 clip_definition = (CLIP_DEFINITION **)trealloc(clip_definition, sizeof(*clip_definition) * (clip_definitions + 1));
726 if (!(clip_definition[clip_definitions] = process_new_clip_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
727 SDDS_Bomb("invalid -clip syntax");
728 add_definition(&definition, &definitions, (void *)clip_definition[clip_definitions], IS_CLIP_DEFINITION);
729 clip_definitions++;
730 break;
731 case SET_FCLIP:
732 fclip_definition = (FCLIP_DEFINITION **)trealloc(fclip_definition, sizeof(*fclip_definition) * (fclip_definitions + 1));
733 if (!(fclip_definition[fclip_definitions] = process_new_fclip_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
734 SDDS_Bomb("invalid -fclip syntax");
735 add_definition(&definition, &definitions, (void *)fclip_definition[fclip_definitions], IS_FCLIP_DEFINITION);
736 fclip_definitions++;
737 break;
738 case SET_SPARSE:
739 sparse_definition = (SPARSE_DEFINITION **)trealloc(sparse_definition, sizeof(*sparse_definition) * (sparse_definitions + 1));
740 if (!(sparse_definition[sparse_definitions] = process_new_sparse_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
741 SDDS_Bomb("invalid -sparse syntax");
742 add_definition(&definition, &definitions, (void *)sparse_definition[sparse_definitions], IS_SPARSE_DEFINITION);
743 sparse_definitions++;
744 break;
745 case SET_SAMPLE:
746 sample_definition = (SAMPLE_DEFINITION **)trealloc(sample_definition, sizeof(*sample_definition) * (sample_definitions + 1));
747 if (!(sample_definition[sample_definitions] = process_new_sample_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
748 SDDS_Bomb("invalid -sample syntax");
749 add_definition(&definition, &definitions, (void *)sample_definition[sample_definitions], IS_SAMPLE_DEFINITION);
750 sample_definitions++;
751 break;
752 case SET_PIPE:
753 if (!processPipeOption(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1, &pipeFlags))
754 SDDS_Bomb("invalid -pipe syntax");
755 break;
756 case SET_FORMAT:
757 if (s_arg[i_arg].n_items < 4)
758 SDDS_Bomb("invalid -format syntax");
759 format_definition = (FORMAT_DEFINITION **)trealloc(format_definition, sizeof(*format_definition) * (format_definitions + 1));
760 if (!(format_definition[format_definitions] = process_new_format_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
761 SDDS_Bomb("invalid -format syntax");
762 add_definition(&definition, &definitions, (void *)format_definition[format_definitions], IS_FORMAT_DEFINITION);
763 format_definitions++;
764 break;
765 case SET_CAST:
766 if (s_arg[i_arg].n_items != 5)
767 SDDS_Bomb("invalid -cast syntax");
768 cast_definition = (CAST_DEFINITION **)trealloc(cast_definition, sizeof(*cast_definition) * (cast_definitions + 1));
769 if (!(cast_definition[cast_definitions] = process_new_cast_definition(s_arg[i_arg].list + 1, s_arg[i_arg].n_items - 1)))
770 SDDS_Bomb("invalid -cast syntax");
771 add_definition(&definition, &definitions, (void *)cast_definition[cast_definitions], IS_CAST_DEFINITION);
772 cast_definitions++;
773 break;
774 case SET_DELETE:
775 if (s_arg[i_arg].n_items < 3)
776 SDDS_Bomb("invalid -delete syntax");
777 switch (match_string(s_arg[i_arg].list[1], mode_name, MODES, 0)) {
778 case COLUMN_MODE:
779 delete_column = trealloc(delete_column, sizeof(*delete_column) * (delete_columns + s_arg[i_arg].n_items - 2));
780 for (i = 2; i < s_arg[i_arg].n_items; i++)
781 delete_column[i - 2 + delete_columns] = s_arg[i_arg].list[i];
782 delete_columns += s_arg[i_arg].n_items - 2;
783 break;
784 case PARAMETER_MODE:
785 delete_parameter = trealloc(delete_parameter, sizeof(*delete_parameter) * (delete_parameters + s_arg[i_arg].n_items - 2));
786 for (i = 2; i < s_arg[i_arg].n_items; i++)
787 delete_parameter[i - 2 + delete_parameters] = s_arg[i_arg].list[i];
788 delete_parameters += s_arg[i_arg].n_items - 2;
789 break;
790 case ARRAY_MODE:
791 delete_array = trealloc(delete_array, sizeof(*delete_array) * (delete_arrays + s_arg[i_arg].n_items - 2));
792 for (i = 2; i < s_arg[i_arg].n_items; i++)
793 delete_array[i - 2 + delete_arrays] = s_arg[i_arg].list[i];
794 delete_arrays += s_arg[i_arg].n_items - 2;
795 break;
796 default:
797 SDDS_Bomb("invalid -delete syntax: specify column or parameter keyword");
798 break;
799 }
800 break;
801 case SET_RETAIN:
802 if (s_arg[i_arg].n_items < 3)
803 SDDS_Bomb("invalid -retain syntax");
804 switch (match_string(s_arg[i_arg].list[1], mode_name, MODES, 0)) {
805 case COLUMN_MODE:
806 retain_column = trealloc(retain_column, sizeof(*retain_column) * (retain_columns + s_arg[i_arg].n_items - 2));
807 for (i = 2; i < s_arg[i_arg].n_items; i++)
808 retain_column[i - 2 + retain_columns] = s_arg[i_arg].list[i];
809 retain_columns += s_arg[i_arg].n_items - 2;
810 break;
811 case PARAMETER_MODE:
812 retain_parameter = trealloc(retain_parameter, sizeof(*retain_parameter) * (retain_parameters + s_arg[i_arg].n_items - 2));
813 for (i = 2; i < s_arg[i_arg].n_items; i++)
814 retain_parameter[i - 2 + retain_parameters] = s_arg[i_arg].list[i];
815 retain_parameters += s_arg[i_arg].n_items - 2;
816 break;
817 case ARRAY_MODE:
818 retain_array = trealloc(retain_array, sizeof(*retain_array) * (retain_arrays + s_arg[i_arg].n_items - 2));
819 for (i = 2; i < s_arg[i_arg].n_items; i++)
820 retain_array[i - 2 + retain_arrays] = s_arg[i_arg].list[i];
821 retain_arrays += s_arg[i_arg].n_items - 2;
822 break;
823 default:
824 SDDS_Bomb("invalid -retain syntax: specify column or parameter keyword");
825 break;
826 }
827 break;
828 case SET_DESCRIPTION:
829 s_arg[i_arg].n_items -= 1;
830 if (!scanItemList(&flags, s_arg[i_arg].list + 1, &s_arg[i_arg].n_items, 0,
831 "text", SDDS_STRING, &description_text, 1, 0,
832 "contents", SDDS_STRING, &description_contents, 1, 0, NULL))
833 SDDS_Bomb("invalid -description syntax");
834 break;
835 case SET_THREADS:
836 if (s_arg[i_arg].n_items != 2 ||
837 !sscanf(s_arg[i_arg].list[1], "%d", &threads) || threads < 1)
838 SDDS_Bomb("invalid -threads syntax");
839 break;
840 default:
841 snprintf(buffer, SDDS_MAXLINE, "unknown switch: %s", s_arg[i_arg].list[0]);
842 SDDS_Bomb(buffer);
843 break;
844 }
845 } else {
846 if (inputfile == NULL)
847 inputfile = s_arg[i_arg].list[0];
848 else if (outputfile == NULL)
849 outputfile = s_arg[i_arg].list[0];
850 else {
851 fprintf(stderr, "argument %s is invalid: too many filenames (sddsprocess)\n", s_arg[i_arg].list[0]);
852 exit(EXIT_FAILURE);
853 }
854 }
855 }
856
857 processFilenames("sddsprocess", &inputfile, &outputfile, pipeFlags, nowarnings, &tmpfile_used);
858
859 if (rpn_definitions_files) {
860 rpn(rpn_definitions_file[0]);
861 if (rpn_check_error())
862 exit(EXIT_FAILURE);
863 for (i = 1; i < rpn_definitions_files; i++) {
864 sprintf(s, "\"%s,s\" @", rpn_definitions_file[i]);
865 rpn(s);
866 if (rpn_check_error())
867 exit(EXIT_FAILURE);
868 }
869 } else {
870 rpn(getenv("RPN_DEFNS"));
871 if (rpn_check_error())
872 exit(EXIT_FAILURE);
873 }
874 i_page_memory = rpn_create_mem("i_page", 0);
875 table_number_memory = rpn_create_mem("table_number", 0);
876 n_rows_memory = rpn_create_mem("n_rows", 0);
877
878 if (!SDDS_InitializeInput(&SDDS_input, inputfile)) {
879 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
880 exit(EXIT_FAILURE);
881 }
882 if (!check_ifitems(&SDDS_input, &ifnot_item, 0, !nowarnings) ||
883 !check_ifitems(&SDDS_input, &ifis_item, 1, !nowarnings))
884 exit(EXIT_SUCCESS);
885 original_parameters = SDDS_input.layout.n_parameters;
886
887 if (!description_text)
888 SDDS_GetDescription(&SDDS_input, &description_text, &description_contents);
889 if (!SDDS_InitializeOutput(&SDDS_output, SDDS_input.layout.data_mode.mode, 1, description_text, description_contents, outputfile)) {
890 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
891 exit(EXIT_FAILURE);
892 }
893 if (columnMajorOrder != -1)
894 SDDS_output.layout.data_mode.column_major = columnMajorOrder;
895 else
896 SDDS_output.layout.data_mode.column_major = SDDS_input.layout.data_mode.column_major;
897 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
898
899#ifdef DEBUG
900 fprintf(stderr, "Getting column, parameter, and array names from input file.\n");
901#endif
902 if (!(orig_column_name = SDDS_GetColumnNames(&SDDS_input, &orig_column_names))) {
903 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
904 exit(EXIT_FAILURE);
905 }
906 if (orig_column_names && !(new_column_name = process_name_options(orig_column_name, &orig_column_flag, orig_column_names, delete_column, delete_columns, retain_column, retain_columns, rename_column, rename_columns, edit_column_request, edit_column_requests)))
907 exit(EXIT_FAILURE);
908
909 if (!(orig_parameter_name = SDDS_GetParameterNames(&SDDS_input, &orig_parameter_names))) {
910 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
911 exit(EXIT_FAILURE);
912 }
913
914#ifdef DEBUG
915 fprintf(stderr, "processing name options\n");
916#endif
917 if (orig_parameter_names && !(new_parameter_name = process_name_options(orig_parameter_name, &orig_parameter_flag, orig_parameter_names, delete_parameter, delete_parameters, retain_parameter, retain_parameters, rename_parameter, rename_parameters, edit_parameter_request, edit_parameter_requests)))
918 exit(EXIT_FAILURE);
919
920 if (!(orig_array_name = SDDS_GetArrayNames(&SDDS_input, &orig_array_names))) {
921 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
922 exit(EXIT_FAILURE);
923 }
924 if (orig_array_names && !(new_array_name = process_name_options(orig_array_name, &orig_array_flag, orig_array_names, delete_array, delete_arrays, retain_array, retain_arrays, rename_array, rename_arrays, edit_array_request, edit_array_requests)))
925 exit(EXIT_FAILURE);
926
927#ifdef DEBUG
928 for (i = 0; i < orig_parameter_names; i++) {
929 fprintf(stderr, "parameter %s goes to %s\n", orig_parameter_name[i], orig_parameter_flag[i] ? new_parameter_name[i] : "<deleted>");
930 }
931 for (i = 0; i < orig_array_names; i++) {
932 fprintf(stderr, "array %s goes to %s\n", orig_array_name[i], orig_array_flag[i] ? new_array_name[i] : "<deleted>");
933 }
934 for (i = 0; i < orig_column_names; i++) {
935 fprintf(stderr, "column %s goes to %s\n", orig_column_name[i], orig_column_flag[i] ? new_column_name[i] : "<deleted>");
936 }
937#endif
938
939#ifdef DEBUG
940 fprintf(stderr, "Transferring definitions to new file.\n");
941#endif
942 for (i = 0; i < orig_parameter_names; i++) {
943 if (orig_parameter_flag[i]) {
944 if (!SDDS_TransferParameterDefinition(&SDDS_output, &SDDS_input, orig_parameter_name[i], new_parameter_name[i])) {
945 fprintf(stderr, "unable to transfer parameter %s to %s\n", orig_parameter_name[i], new_parameter_name[i]);
946 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
947 exit(EXIT_FAILURE);
948 }
949 }
950 }
951 for (i = 0; i < orig_array_names; i++) {
952 if (orig_array_flag[i]) {
953 if (!SDDS_TransferArrayDefinition(&SDDS_output, &SDDS_input, orig_array_name[i], new_array_name[i])) {
954 fprintf(stderr, "unable to transfer array %s to %s\n", orig_array_name[i], new_array_name[i]);
955 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
956 exit(EXIT_FAILURE);
957 }
958 }
959 }
960 //output_columns = 0;
961 for (i = 0; i < orig_column_names; i++) {
962 if (orig_column_flag[i]) {
963 //output_columns++;
964 if (!SDDS_TransferColumnDefinition(&SDDS_output, &SDDS_input, orig_column_name[i], new_column_name[i])) {
965 fprintf(stderr, "unable to transfer column %s to %s\n", orig_column_name[i], new_column_name[i]);
966 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
967 exit(EXIT_FAILURE);
968 }
969 }
970 }
971 for (i = 0; i < orig_column_names; i++) {
972 free(new_column_name[i]);
973 free(orig_column_name[i]);
974 }
975 free(new_column_name);
976 free(orig_column_name);
977
978 expandDefinitions(&definition, &definitions, &SDDS_output);
979
980 for (i = 0; i < definitions; i++) {
981 switch (definition[i].type) {
982 case IS_EQUATION_DEFINITION:
983 equation_ptr = (EQUATION_DEFINITION *)definition[i].structure;
984 sprintf(s, "%sUDF%ld", equation_ptr->name, udfCounter++);
985 delete_chars(s, " ");
986 if (!SDDS_CopyString(&equation_ptr->udf_name, s))
987 SDDS_Bomb("string copying failed (making udf name)");
988 if (equation_ptr->equation[0] == '@') {
989 if (equation_ptr->equation[1] == '@') {
990 /* parameter contains algebraic notation */
991 if ((index = SDDS_GetParameterIndex(&SDDS_output, equation_ptr->equation + 2)) < 0 ||
992 SDDS_GetParameterType(&SDDS_input, index) != SDDS_STRING) {
993 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
994 fprintf(stderr, "Error (sddsprocess): parameter %s does not exist or is not string type\n", equation_ptr->equation + 2);
995 exit(EXIT_FAILURE);
996 }
997 } else {
998 if ((index = SDDS_GetParameterIndex(&SDDS_output, equation_ptr->equation + 1)) < 0 ||
999 SDDS_GetParameterType(&SDDS_input, index) != SDDS_STRING) {
1000 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1001 fprintf(stderr, "Error (sddsprocess): parameter %s does not exist or is not string type\n", equation_ptr->equation + 1);
1002 exit(EXIT_FAILURE);
1003 }
1004 }
1005 } else {
1006 if (strstr(equation_ptr->equation, "xstr") == NULL)
1007 create_udf(equation_ptr->udf_name, equation_ptr->equation);
1008 else
1009 /* This works around a bug in rpn that results in a stack overflow for definitions that use the 'xstr' function */
1010 equation_ptr->udf_name = equation_ptr->equation;
1011 }
1012 if (equation_ptr->redefinition) {
1013 if (equation_ptr->is_parameter) {
1014 if ((index = SDDS_GetParameterIndex(&SDDS_output, equation_ptr->name)) < 0)
1015 equation_ptr->redefinition = 0;
1016 } else if ((index = SDDS_GetColumnIndex(&SDDS_output, equation_ptr->name)) < 0)
1017 equation_ptr->redefinition = 0;
1018 }
1019 if (equation_ptr->redefinition == 0) {
1020 if (equation_ptr->is_parameter) {
1021 if (!SDDS_ProcessParameterString(&SDDS_output, equation_ptr->text, SDDS_WRITEONLY_DEFINITION)) {
1022 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1023 fprintf(stderr, "Bad parameter string: %s\n", equation_ptr->text);
1024 exit(EXIT_FAILURE);
1025 }
1026 } else if (!SDDS_ProcessColumnString(&SDDS_output, equation_ptr->text, SDDS_WRITEONLY_DEFINITION)) {
1027 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1028 fprintf(stderr, "Bad column string: %s\n", equation_ptr->text);
1029 exit(EXIT_FAILURE);
1030 }
1031 } else {
1032 if (equation_ptr->is_parameter) {
1033 if (!REDEFINABLE_TYPE(SDDS_GetParameterType(&SDDS_output, index))) {
1034 fprintf(stderr, "can't redefine parameter %s--non-numeric\n", equation_ptr->name);
1035 exit(EXIT_FAILURE);
1036 }
1037 if (equation_ptr->argc && !SDDS_RedefineParameterCL(&SDDS_output, equation_ptr->name, equation_ptr->argv, equation_ptr->argc)) {
1038 fprintf(stderr, "can't redefine parameter %s as requested\n", equation_ptr->name);
1039 exit(EXIT_FAILURE);
1040 }
1041 } else {
1042 if (!REDEFINABLE_TYPE(SDDS_GetColumnType(&SDDS_output, index))) {
1043 fprintf(stderr, "can't redefine column %s--non-numeric\n", equation_ptr->name);
1044 exit(EXIT_FAILURE);
1045 }
1046 if (equation_ptr->argc && !SDDS_RedefineColumnCL(&SDDS_output, equation_ptr->name, equation_ptr->argv, equation_ptr->argc)) {
1047 fprintf(stderr, "can't redefine column %s as requested\n", equation_ptr->name);
1048 exit(EXIT_FAILURE);
1049 }
1050 }
1051 }
1052 break;
1053 case IS_EDIT_DEFINITION:
1054 edit_ptr = (EDIT_DEFINITION *)definition[i].structure;
1055 index = edit_ptr->is_parameter ? SDDS_GetParameterIndex(&SDDS_output, edit_ptr->new_name) : SDDS_GetColumnIndex(&SDDS_output, edit_ptr->new_name);
1056 if (index < 0)
1057 edit_ptr->reedit = 0;
1058 else if (!edit_ptr->reedit) {
1059 fprintf(stderr, "Can't define new -edit %s %s--already exists (sddsprocess)\n", edit_ptr->is_parameter ? "parameter" : "column", edit_ptr->new_name);
1060 exit(EXIT_FAILURE);
1061 }
1062 if (!edit_ptr->reedit) {
1063 if (!(edit_ptr->is_parameter ? SDDS_ProcessParameterString(&SDDS_output, edit_ptr->text, SDDS_WRITEONLY_DEFINITION) : SDDS_ProcessColumnString(&SDDS_output, edit_ptr->text, SDDS_WRITEONLY_DEFINITION))) {
1064 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1065 exit(EXIT_FAILURE);
1066 }
1067 } else {
1068 if ((edit_ptr->is_parameter ? SDDS_GetParameterType(&SDDS_output, index) : SDDS_GetColumnType(&SDDS_output, index)) != SDDS_STRING) {
1069 fprintf(stderr, "Can't reedit %s %s--not string type (sddsprocess)\n", edit_ptr->is_parameter ? "parameter" : "column", edit_ptr->new_name);
1070 exit(EXIT_FAILURE);
1071 }
1072 if (edit_ptr->argc && !(edit_ptr->is_parameter ? SDDS_RedefineParameterCL(&SDDS_output, edit_ptr->new_name, edit_ptr->argv, edit_ptr->argc) : SDDS_RedefineColumnCL(&SDDS_output, edit_ptr->new_name, edit_ptr->argv, edit_ptr->argc))) {
1073 fprintf(stderr, "Can't reedit %s %s as requested (sddsprocess)\n", edit_ptr->is_parameter ? "parameter" : "column", edit_ptr->new_name);
1074 exit(EXIT_FAILURE);
1075 }
1076 }
1077 break;
1078 case IS_SCAN_DEFINITION:
1079 scan_ptr = (SCAN_DEFINITION *)definition[i].structure;
1080 if (scan_ptr->is_parameter) {
1081 if (!SDDS_ProcessParameterString(&SDDS_output, scan_ptr->text, SDDS_WRITEONLY_DEFINITION)) {
1082 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1083 exit(EXIT_FAILURE);
1084 }
1085 } else if (!SDDS_ProcessColumnString(&SDDS_output, scan_ptr->text, SDDS_WRITEONLY_DEFINITION)) {
1086 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1087 exit(EXIT_FAILURE);
1088 }
1089 break;
1090 case IS_PRINT_DEFINITION:
1091 print_ptr = (PRINT_DEFINITION *)definition[i].structure;
1092 index = print_ptr->is_parameter ? SDDS_GetParameterIndex(&SDDS_output, print_ptr->new_name) : SDDS_GetColumnIndex(&SDDS_output, print_ptr->new_name);
1093 if (index < 0)
1094 print_ptr->reprint = 0;
1095 else if (!print_ptr->reprint) {
1096 fprintf(stderr, "Can't define new -print %s %s--already exists (sddsprocess)\n", print_ptr->is_parameter ? "parameter" : "column", print_ptr->new_name);
1097 exit(EXIT_FAILURE);
1098 }
1099 if (!print_ptr->reprint) {
1100 if (!(print_ptr->is_parameter ? SDDS_ProcessParameterString(&SDDS_output, print_ptr->text, SDDS_WRITEONLY_DEFINITION) : SDDS_ProcessColumnString(&SDDS_output, print_ptr->text, SDDS_WRITEONLY_DEFINITION))) {
1101 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1102 exit(EXIT_FAILURE);
1103 }
1104 } else {
1105 if ((print_ptr->is_parameter ? SDDS_GetParameterType(&SDDS_output, index) : SDDS_GetColumnType(&SDDS_output, index)) != SDDS_STRING) {
1106 fprintf(stderr, "Can't reprint %s %s--not string type (sddsprocess)\n", print_ptr->is_parameter ? "parameter" : "column", print_ptr->new_name);
1107 exit(EXIT_FAILURE);
1108 }
1109 }
1110 break;
1111 case IS_PROCESSING_DEFINITION:
1112 processing_ptr = (PROCESSING_DEFINITION *)definition[i].structure;
1113 if (!complete_processing_definitions(&processing_ptr, 1, &SDDS_output)) {
1114 fprintf(stderr, "error: problem completing processing definition\n");
1115 if (SDDS_NumberOfErrors())
1116 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1117 exit(EXIT_FAILURE);
1118 }
1119 break;
1120 case IS_CONVERSION_DEFINITION:
1121 conversion_ptr = (CONVERSION_DEFINITION *)definition[i].structure;
1122 if (conversion_ptr->is_parameter) {
1123 if ((j = SDDS_GetParameterIndex(&SDDS_output, conversion_ptr->name)) < 0)
1124 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors | SDDS_EXIT_PrintErrors);
1125 if (SDDS_CheckParameter(&SDDS_output, conversion_ptr->name, conversion_ptr->old_units, SDDS_GetParameterType(&SDDS_output, j), stderr) != SDDS_CHECK_OKAY)
1126 exit(EXIT_FAILURE);
1127 if (!SDDS_ChangeParameterInformation(&SDDS_output, "units", conversion_ptr->new_units, SDDS_SET_BY_NAME, conversion_ptr->name)) {
1128 if (SDDS_NumberOfErrors())
1129 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1130 else
1131 fprintf(stderr, "error: unknown error processing conversion for parameter %s\n", conversion_ptr->name);
1132 exit(EXIT_FAILURE);
1133 }
1134 } else {
1135 if ((j = SDDS_GetColumnIndex(&SDDS_output, conversion_ptr->name)) < 0)
1136 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors | SDDS_EXIT_PrintErrors);
1137 if (SDDS_CheckColumn(&SDDS_output, conversion_ptr->name, conversion_ptr->old_units, SDDS_GetColumnType(&SDDS_output, j), stderr) != SDDS_CHECK_OKAY)
1138 exit(EXIT_FAILURE);
1139 if (!SDDS_ChangeColumnInformation(&SDDS_output, "units", conversion_ptr->new_units, SDDS_SET_BY_NAME, conversion_ptr->name)) {
1140 if (SDDS_NumberOfErrors())
1141 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1142 else
1143 fprintf(stderr, "error: unknown error processing conversion for column %s\n", conversion_ptr->name);
1144 exit(EXIT_FAILURE);
1145 }
1146 }
1147 break;
1148 case IS_SYSTEM_DEFINITION:
1149 system_ptr = (SYSTEM_DEFINITION *)definition[i].structure;
1150 if (system_ptr->is_parameter) {
1151 if (!SDDS_ProcessParameterString(&SDDS_output, system_ptr->text, SDDS_WRITEONLY_DEFINITION)) {
1152 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1153 exit(EXIT_FAILURE);
1154 }
1155 } else if (!SDDS_ProcessColumnString(&SDDS_output, system_ptr->text, SDDS_WRITEONLY_DEFINITION)) {
1156 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1157 exit(EXIT_FAILURE);
1158 }
1159 break;
1160 case IS_FORMAT_DEFINITION:
1161 format_ptr = (FORMAT_DEFINITION *)definition[i].structure;
1162 if (strcmp(format_ptr->source, format_ptr->target)) {
1163 if (format_ptr->is_parameter) {
1164 if (!SDDS_TransferParameterDefinition(&SDDS_output, &SDDS_output, format_ptr->source, format_ptr->target)) {
1165 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1166 exit(EXIT_FAILURE);
1167 }
1168 } else if (!SDDS_TransferColumnDefinition(&SDDS_output, &SDDS_output, format_ptr->source, format_ptr->target)) {
1169 fprintf(stderr, "Problem setting up to format %s from %s\n", format_ptr->target, format_ptr->source);
1170 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1171 exit(EXIT_FAILURE);
1172 }
1173 }
1174 break;
1175 case IS_EVALUATE_DEFINITION:
1176 evaluate_ptr = (EVALUATE_DEFINITION *)definition[i].structure;
1177 index = evaluate_ptr->is_parameter ? SDDS_GetParameterIndex(&SDDS_output, evaluate_ptr->name) : SDDS_GetColumnIndex(&SDDS_output, evaluate_ptr->name);
1178 if (index >= 0) {
1179 fprintf(stderr, "Can't define new -evaluate %s %s--already exists (sddsprocess)\n", evaluate_ptr->is_parameter ? "parameter" : "column", evaluate_ptr->name);
1180 exit(EXIT_FAILURE);
1181 }
1182 if ((index = evaluate_ptr->is_parameter ? SDDS_GetParameterIndex(&SDDS_output, evaluate_ptr->source) : SDDS_GetColumnIndex(&SDDS_output, evaluate_ptr->source)) < 0) {
1183 fprintf(stderr, "Can't evaluate %s %s--doesn't exist (sddsprocess)\n", evaluate_ptr->is_parameter ? "parameter" : "column", evaluate_ptr->source);
1184 exit(EXIT_FAILURE);
1185 }
1186 if ((evaluate_ptr->is_parameter ? SDDS_GetParameterType(&SDDS_output, index) : SDDS_GetColumnType(&SDDS_output, index)) != SDDS_STRING) {
1187 fprintf(stderr, "Can't evaluate %s %s--not string type (sddsprocess)\n", evaluate_ptr->is_parameter ? "parameter" : "column", evaluate_ptr->source);
1188 exit(EXIT_FAILURE);
1189 }
1190 if (!(evaluate_ptr->is_parameter ? SDDS_ProcessParameterString(&SDDS_output, evaluate_ptr->text, SDDS_WRITEONLY_DEFINITION) : SDDS_ProcessColumnString(&SDDS_output, evaluate_ptr->text, SDDS_WRITEONLY_DEFINITION))) {
1191 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1192 exit(EXIT_FAILURE);
1193 }
1194 break;
1195 case IS_FILTER_DEFINITION:
1196 case IS_MATCH_DEFINITION:
1197 case IS_RPNTEST_DEFINITION:
1198 case IS_RPNEXPRESSION_DEFINITION:
1199 case IS_CLIP_DEFINITION:
1200 case IS_FCLIP_DEFINITION:
1201 case IS_SPARSE_DEFINITION:
1202 case IS_SAMPLE_DEFINITION:
1203 case IS_NUMBERTEST_DEFINITION:
1204 case IS_TIME_FILTER_DEFINITION:
1205 break;
1206 case IS_CAST_DEFINITION:
1207 if (!complete_cast_definition(&SDDS_output, (CAST_DEFINITION *)definition[i].structure, &SDDS_input)) {
1208 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1209 exit(EXIT_FAILURE);
1210 }
1211 break;
1212 default:
1213 fprintf(stderr, "error: unknown definition type code %ld seen--this shouldn't happen!\n", definition[i].type);
1214 exit(EXIT_FAILURE);
1215 break;
1216 }
1217 }
1218
1219 if (!SDDS_DeleteParameterFixedValues(&SDDS_output)) {
1220 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1221 exit(EXIT_FAILURE);
1222 }
1223
1224 if (!SDDS_WriteLayout(&SDDS_output)) {
1225 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1226 exit(EXIT_FAILURE);
1227 }
1228
1229 if (summarize) {
1230 for (i = 0; i < definitions; i++) {
1231 switch (definition[i].type) {
1232 case IS_EQUATION_DEFINITION:
1233 equation_ptr = (EQUATION_DEFINITION *)definition[i].structure;
1234 fprintf(stderr, "%s %s created from equation %s:\n %s\n", equation_ptr->is_parameter ? "parameter" : "column", equation_ptr->name, equation_ptr->equation, equation_ptr->text);
1235 break;
1236 case IS_EVALUATE_DEFINITION:
1237 evaluate_ptr = (EVALUATE_DEFINITION *)definition[i].structure;
1238 fprintf(stderr, "evaluating to %s %s from %s\n", evaluate_ptr->is_parameter ? "parameter" : "column", evaluate_ptr->name, evaluate_ptr->source);
1239 break;
1240 case IS_EDIT_DEFINITION:
1241 edit_ptr = (EDIT_DEFINITION *)definition[i].structure;
1242 fprintf(stderr, "%s %s edit from %s using %s:\n %s\n", edit_ptr->is_parameter ? "parameter" : "column", edit_ptr->new_name, edit_ptr->source, edit_ptr->edit_command, edit_ptr->text);
1243 break;
1244 case IS_SCAN_DEFINITION:
1245 scan_ptr = (SCAN_DEFINITION *)definition[i].structure;
1246 fprintf(stderr, "%s %s scanned from %s using %s:\n %s\n", scan_ptr->is_parameter ? "parameter" : "column", scan_ptr->new_name, scan_ptr->source, scan_ptr->sscanf_string, scan_ptr->text);
1247 break;
1248 case IS_PRINT_DEFINITION:
1249 print_ptr = (PRINT_DEFINITION *)definition[i].structure;
1250 fprintf(stderr, "printing to %s %s from ", print_ptr->is_parameter ? "parameter" : "column", print_ptr->new_name);
1251 for (j = 0; j < print_ptr->sources; j++)
1252 fprintf(stderr, "%s ", print_ptr->source[j]);
1253 fprintf(stderr, ":\n %s\n", print_ptr->text);
1254 break;
1255 case IS_PROCESSING_DEFINITION:
1256 processing_ptr = (PROCESSING_DEFINITION *)definition[i].structure;
1257 fprintf(stderr, "column %s will be processed to give %s in parameter %s\n", processing_ptr->column_name, processing_ptr->description, processing_ptr->parameter_name);
1258 break;
1259 case IS_CONVERSION_DEFINITION:
1260 conversion_ptr = (CONVERSION_DEFINITION *)definition[i].structure;
1261 fprintf(stderr, "%s %s will be converted to %s using factor %e\n", conversion_ptr->is_parameter ? "parameter" : "column", conversion_ptr->name, conversion_ptr->new_units, conversion_ptr->factor);
1262 break;
1263 case IS_FILTER_DEFINITION:
1264 filter_ptr = (FILTER_DEFINITION *)definition[i].structure;
1265 show_filters(filter_ptr->is_parameter ? "parameter" : "column", filter_ptr->filter_term, filter_ptr->filter_terms);
1266 break;
1267 case IS_TIME_FILTER_DEFINITION:
1268 time_filter_ptr = (TIME_FILTER_DEFINITION *)definition[i].structure;
1269 fprintf(stderr, "Time filter: <%s> %22.15e, %22.15e\n", time_filter_ptr->name, time_filter_ptr->before, time_filter_ptr->after);
1270 case IS_MATCH_DEFINITION:
1271 match_ptr = (MATCH_DEFINITION *)definition[i].structure;
1272 show_matches(match_ptr->is_parameter ? "parameter" : "column", match_ptr->match_term, match_ptr->match_terms);
1273 break;
1274 case IS_RPNTEST_DEFINITION:
1275 rpntest_ptr = (RPNTEST_DEFINITION *)definition[i].structure;
1276 if (rpntest_ptr->autostop)
1277 fprintf(stderr, "%s-based autostop rpn test:\n %s\n", rpntest_ptr->is_parameter ? "parameter" : "column", rpntest_ptr->expression);
1278 else
1279 fprintf(stderr, "%s-based rpn test:\n %s\n", rpntest_ptr->is_parameter ? "parameter" : "column", rpntest_ptr->expression);
1280 break;
1281 case IS_NUMBERTEST_DEFINITION:
1282 numbertest_ptr = (NUMBERTEST_DEFINITION *)definition[i].structure;
1283 fprintf(stderr, "%s-based number test:\n %s %s%s\n", numbertest_ptr->is_parameter ? "parameter" : "column", numbertest_ptr->name, numbertest_ptr->flags & NUMSCANFILTER_INVERT ? " inverted" : "", numbertest_ptr->flags & NUMSCANFILTER_STRICT ? " strict" : "");
1284 break;
1285 case IS_RPNEXPRESSION_DEFINITION:
1286 rpnexpression_ptr = (RPNEXPRESSION_DEFINITION *)definition[i].structure;
1287 fprintf(stderr, "%srepeated rpn expression:\n %s\n", (rpnexpression_ptr->repeat ? "" : "un"), rpnexpression_ptr->expression);
1288 break;
1289 case IS_SYSTEM_DEFINITION:
1290 system_ptr = (SYSTEM_DEFINITION *)definition[i].structure;
1291 fprintf(stderr, "%s %s system-call from %s:\n %s\n", system_ptr->is_parameter ? "parameter" : "column", system_ptr->new_name, system_ptr->source, system_ptr->text);
1292 break;
1293 case IS_FORMAT_DEFINITION:
1294 format_ptr = (FORMAT_DEFINITION *)definition[i].structure;
1295 fprintf(stderr, "formating %s %s from %s using ", format_ptr->is_parameter ? "parameter" : "column", format_ptr->target, format_ptr->source);
1296 fprintf(stderr, "string: %s double: %s long: %s\n", format_ptr->stringFormat ? format_ptr->stringFormat : "<none>", format_ptr->doubleFormat ? format_ptr->doubleFormat : "<none>", format_ptr->longFormat ? format_ptr->longFormat : "<none>");
1297 break;
1298 case IS_CLIP_DEFINITION:
1299 clip_ptr = (CLIP_DEFINITION *)definition[i].structure;
1300 fprintf(stderr, "page clip operation:\n head = %" PRId64 " tail = %" PRId64 " invert = %hd\n", clip_ptr->head, clip_ptr->tail, clip_ptr->invert);
1301 break;
1302 case IS_FCLIP_DEFINITION:
1303 fclip_ptr = (FCLIP_DEFINITION *)definition[i].structure;
1304 fprintf(stderr, "page clip operation:\n head = %f tail = %f invert = %hd\n", fclip_ptr->fhead, fclip_ptr->ftail, fclip_ptr->invert);
1305 break;
1306 case IS_SPARSE_DEFINITION:
1307 sparse_ptr = (SPARSE_DEFINITION *)definition[i].structure;
1308 fprintf(stderr, "page sparse operation:\n interval = %" PRId64 " offset = %" PRId64 "\n", sparse_ptr->interval, sparse_ptr->offset);
1309 break;
1310 case IS_SAMPLE_DEFINITION:
1311 sample_ptr = (SAMPLE_DEFINITION *)definition[i].structure;
1312 fprintf(stderr, "page sample operation:\n fraction = %.8g\n", sample_ptr->fraction);
1313 break;
1314 case IS_CAST_DEFINITION:
1315 cast_ptr = (CAST_DEFINITION *)definition[i].structure;
1316 fprintf(stderr, "%s %s cast from %s to type %s\n", cast_ptr->isParameter ? "parameter" : "column", cast_ptr->newName, cast_ptr->source, cast_ptr->newTypeName);
1317 break;
1318 default:
1319 fprintf(stderr, "error: unknown definition type code %ld seen--this shouldn't happen!\n", definition[i].type);
1320 exit(EXIT_FAILURE);
1321 break;
1322 }
1323 }
1324 }
1325
1326 while ((page_number = SDDS_ReadPage(&SDDS_input)) > 0) {
1327 if (!SDDS_CopyPage(&SDDS_output, &SDDS_input)) {
1328 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1329 exit(EXIT_FAILURE);
1330 }
1331 if (!SDDS_SetRowFlags(&SDDS_output, 1)) {
1332 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1333 exit(EXIT_FAILURE);
1334 }
1335 if ((n_rows = SDDS_CountRowsOfInterest(&SDDS_output)) <= 0 && SDDS_ColumnCount(&SDDS_output) != 0) {
1336 if (!nowarnings)
1337 fprintf(stderr, "warning: no rows selected for page %" PRId32 "\n", SDDS_output.page_number);
1338 if (n_rows == 0)
1339 SDDS_ClearErrors();
1340 else
1341 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors | SDDS_EXIT_PrintErrors);
1342 }
1343 if (verbose) {
1344 fprintf(stderr, "page number %ld read in, with %" PRId64 " rows of data\n", page_number, n_rows);
1345 for (i = 0; i < original_parameters; i++) {
1346 SDDS_PrintTypedValue(SDDS_output.parameter[i], 0, SDDS_output.layout.parameter_definition[i].type, NULL, stderr, 0);
1347 fputc('\n', stderr);
1348 }
1349 }
1350 rpn_store((double)page_number, NULL, table_number_memory);
1351 rpn_store((double)page_number, NULL, i_page_memory);
1352 row_deletion = skip_page = 0;
1353 i = -1;
1354 while (++i < definitions && !skip_page) {
1355 rpn_store((double)n_rows, NULL, n_rows_memory);
1356 switch (definition[i].type) {
1357 case IS_CONVERSION_DEFINITION:
1358 conversion_ptr = (CONVERSION_DEFINITION *)definition[i].structure;
1359 if (conversion_ptr->is_parameter) {
1360 if (!SDDS_ApplyFactorToParameter(&SDDS_output, conversion_ptr->name, conversion_ptr->factor)) {
1361 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1362 exit(EXIT_FAILURE);
1363 }
1364 } else if (n_rows && !SDDS_ApplyFactorToColumn(&SDDS_output, conversion_ptr->name, conversion_ptr->factor)) {
1365 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1366 exit(EXIT_FAILURE);
1367 }
1368 if (verbose)
1369 fprintf(stderr, "conversion done: %s %s from %s to %s using factor %e\n", conversion_ptr->is_parameter ? "parameter" : "column", conversion_ptr->name, conversion_ptr->old_units, conversion_ptr->new_units, conversion_ptr->factor);
1370 break;
1371 case IS_PROCESSING_DEFINITION:
1372 processing_ptr = (PROCESSING_DEFINITION *)definition[i].structure;
1373 if (processing_ptr->type != SDDS_STRING) {
1374 if (!process_column(&SDDS_output, processing_ptr, &result, &stringResult, !nowarnings, threads))
1375 exit(EXIT_FAILURE);
1376 if (processing_ptr->outputType != SDDS_STRING) {
1377 if (!SDDS_SetParameters(&SDDS_output, SDDS_SET_BY_NAME | SDDS_PASS_BY_VALUE, processing_ptr->parameter_name, result, NULL)) {
1378 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1379 exit(EXIT_FAILURE);
1380 }
1381 rpn_store(result, NULL, processing_ptr->memory_number);
1382 if (verbose)
1383 fprintf(stderr, "processing result: %s = %e\n", processing_ptr->parameter_name, result);
1384 } else {
1385 if (!SDDS_SetParameters(&SDDS_output, SDDS_SET_BY_NAME | SDDS_PASS_BY_VALUE, processing_ptr->parameter_name, stringResult, NULL)) {
1386 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1387 exit(EXIT_FAILURE);
1388 }
1389 if (verbose)
1390 fprintf(stderr, "processing result: %s = %s\n", processing_ptr->parameter_name, stringResult);
1391 free(stringResult);
1392 }
1393 } else {
1394 if (!(stringResult = process_string_column(&SDDS_output, processing_ptr, !nowarnings)) ||
1395 !SDDS_SetParameters(&SDDS_output, SDDS_SET_BY_NAME | SDDS_PASS_BY_VALUE, processing_ptr->parameter_name, stringResult, NULL)) {
1396 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1397 exit(EXIT_FAILURE);
1398 }
1399 free(stringResult);
1400 }
1401 break;
1402 case IS_EDIT_DEFINITION:
1403 edit_ptr = (EDIT_DEFINITION *)definition[i].structure;
1404 if (edit_ptr->is_parameter) {
1405 if (!edit_parameter_value(&SDDS_output, edit_ptr->new_name, edit_ptr->source, edit_ptr->edit_command)) {
1406 fprintf(stderr, "couldn't edit %s to make %s\n", edit_ptr->source, edit_ptr->new_name);
1407 exit(EXIT_FAILURE);
1408 }
1409 } else if (n_rows && !edit_column_value(&SDDS_output, edit_ptr->new_name, edit_ptr->source, edit_ptr->edit_command)) {
1410 fprintf(stderr, "couldn't edit %s to make %s\n", edit_ptr->source, edit_ptr->new_name);
1411 exit(EXIT_FAILURE);
1412 }
1413 if (verbose)
1414 fprintf(stderr, "edited %s %s to produce %s, using edit-command %s\n", edit_ptr->is_parameter ? "parameter" : "column", edit_ptr->source, edit_ptr->new_name, edit_ptr->edit_command);
1415 break;
1416 case IS_SCAN_DEFINITION:
1417 scan_ptr = (SCAN_DEFINITION *)definition[i].structure;
1418 if (scan_ptr->is_parameter) {
1419 if (!scan_parameter_value(&SDDS_output, scan_ptr->new_name, scan_ptr->source, scan_ptr->sscanf_string, scan_ptr->edit)) {
1420 fprintf(stderr, "couldn't scan %s to make %s\n", scan_ptr->source, scan_ptr->new_name);
1421 exit(EXIT_FAILURE);
1422 }
1423 } else if (n_rows && !scan_column_value(&SDDS_output, scan_ptr->new_name, scan_ptr->source, scan_ptr->sscanf_string, scan_ptr->edit)) {
1424 fprintf(stderr, "couldn't scan %s to make %s\n", scan_ptr->source, scan_ptr->new_name);
1425 exit(EXIT_FAILURE);
1426 }
1427 if (verbose)
1428 fprintf(stderr, "scanned %s %s to produce %s, using sscanf string %s\n", scan_ptr->is_parameter ? "parameter" : "column", scan_ptr->source, scan_ptr->new_name, scan_ptr->sscanf_string);
1429 break;
1430 case IS_PRINT_DEFINITION:
1431 print_ptr = (PRINT_DEFINITION *)definition[i].structure;
1432 if (print_ptr->is_parameter) {
1433 if (!print_parameter_value(&SDDS_output, print_ptr->new_name, print_ptr->source, print_ptr->sources, print_ptr->printf_string)) {
1434 fprintf(stderr, "couldn't print to make parameter %s\n", print_ptr->new_name);
1435 exit(EXIT_FAILURE);
1436 }
1437 } else if (n_rows && !print_column_value(&SDDS_output, print_ptr->new_name, print_ptr->source, print_ptr->sources, print_ptr->printf_string)) {
1438 fprintf(stderr, "couldn't print to make column %s\n", print_ptr->new_name);
1439 exit(EXIT_FAILURE);
1440 }
1441 break;
1442 case IS_EQUATION_DEFINITION:
1443 equation_ptr = (EQUATION_DEFINITION *)definition[i].structure;
1444 if (equation_ptr->is_parameter) {
1445 if (!SDDS_ComputeParameter(&SDDS_output, SDDS_GetParameterIndex(&SDDS_output, equation_ptr->name), equation_ptr->udf_name)) {
1446 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1447 exit(EXIT_FAILURE);
1448 }
1449 } else if (n_rows) {
1450 k = i++;
1451 while (i < definitions && definition[i].type == IS_EQUATION_DEFINITION) {
1452 equation_ptr = (EQUATION_DEFINITION *)definition[i].structure;
1453 if (equation_ptr->is_parameter)
1454 break;
1455 else
1456 i++;
1457 }
1458 if (!SDDS_ComputeSetOfColumns(&SDDS_output, k, i, definition)) {
1459 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1460 exit(EXIT_FAILURE);
1461 }
1462 i--;
1463 }
1464 if (verbose)
1465 fprintf(stderr, "computed %s %s using expression %s\n", equation_ptr->is_parameter ? "parameter" : "column", equation_ptr->name, equation_ptr->equation);
1466 break;
1467 case IS_EVALUATE_DEFINITION:
1468 evaluate_ptr = (EVALUATE_DEFINITION *)definition[i].structure;
1469 if (evaluate_ptr->is_parameter) {
1470 if (!SDDS_EvaluateParameter(&SDDS_output, evaluate_ptr)) {
1471 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1472 exit(EXIT_FAILURE);
1473 }
1474 } else if (n_rows) {
1475 if (!SDDS_EvaluateColumn(&SDDS_output, evaluate_ptr)) {
1476 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1477 exit(EXIT_FAILURE);
1478 }
1479 }
1480 if (verbose)
1481 fprintf(stderr, "computed %s %s using expressions from %s\n", evaluate_ptr->is_parameter ? "parameter" : "column", evaluate_ptr->name, evaluate_ptr->source);
1482 break;
1483 case IS_FORMAT_DEFINITION:
1484 format_ptr = (FORMAT_DEFINITION *)definition[i].structure;
1485 if (format_ptr->is_parameter) {
1486 if (!format_parameter_value(&SDDS_output, format_ptr)) {
1487 fprintf(stderr, "couldn't format parameter %s\n", format_ptr->target);
1488 exit(EXIT_FAILURE);
1489 }
1490 } else if (n_rows && !format_column_value(&SDDS_output, format_ptr)) {
1491 fprintf(stderr, "couldn't format column %s\n", format_ptr->target);
1492 exit(EXIT_FAILURE);
1493 }
1494 if (verbose)
1495 fprintf(stderr, "formatted %s %s\n", format_ptr->is_parameter ? "parameter" : "column", format_ptr->target);
1496 break;
1497 case IS_TIME_FILTER_DEFINITION:
1498 time_filter_ptr = (TIME_FILTER_DEFINITION *)definition[i].structure;
1499 if (time_filter_ptr->is_parameter) {
1500 if (!(pardefptr = SDDS_GetParameterDefinition(&SDDS_output, time_filter_ptr->name)) ||
1501 (pardefptr->type == SDDS_STRING || pardefptr->type == SDDS_CHARACTER)) {
1502 fprintf(stderr, "error: unknown or non-numeric parameter %s given for time filter\n", time_filter_ptr->name);
1503 exit(EXIT_FAILURE);
1504 }
1505 accept = SDDS_ItemInsideWindow(SDDS_GetParameter(&SDDS_output, time_filter_ptr->name, &result), 0, pardefptr->type, time_filter_ptr->after, time_filter_ptr->before);
1506
1507 if (!accept)
1508 skip_page = 1;
1509 if (time_filter_ptr->flags & TIMEFILTER_INVERT_GIVEN)
1510 skip_page = !skip_page;
1511 if (skip_page) {
1512 if (verbose)
1513 fputs(" * page outside parameter filter--continuing to next page", stderr);
1514 continue;
1515 }
1516 } else if (n_rows) {
1517 int64_t nleft;
1518 if (verbose) {
1519 fprintf(stderr, " * applying time filter (column %s)", time_filter_ptr->name);
1520 fflush(stderr);
1521 }
1522 if ((nleft = SDDS_FilterRowsOfInterest(&SDDS_output, time_filter_ptr->name, time_filter_ptr->after, time_filter_ptr->before, time_filter_ptr->flags & TIMEFILTER_INVERT_GIVEN ? SDDS_NEGATE_EXPRESSION : SDDS_AND)) < 0) {
1523 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1524 exit(EXIT_FAILURE);
1525 }
1526 if (verbose) {
1527 fprintf(stderr, "---%" PRId64 " rows left\n", nleft);
1528 fflush(stderr);
1529 }
1530 row_deletion = 1;
1531 }
1532 break;
1533 case IS_FILTER_DEFINITION:
1534 filter_ptr = (FILTER_DEFINITION *)definition[i].structure;
1535 filter_term = filter_ptr->filter_term;
1536 for (j = 0; j < filter_ptr->filter_terms; j++) {
1537 if (filter_term[j].lowerPar && !SDDS_GetParameterAsDouble(&SDDS_input, filter_term[j].lowerPar, &(filter_term[j].lower))) {
1538 fprintf(stderr, "Error: unable to get parameter %s value for the lower filter from input file.\n", filter_term[j].lowerPar);
1539 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1540 exit(EXIT_FAILURE);
1541 }
1542 if (filter_term[j].upperPar && !SDDS_GetParameterAsDouble(&SDDS_input, filter_term[j].upperPar, &(filter_term[j].upper))) {
1543 fprintf(stderr, "Error: unable to get parameter %s value for the upper filter from input file.\n", filter_term[j].upperPar);
1544 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1545 exit(EXIT_FAILURE);
1546 }
1547 if (filter_term[j].upper < filter_term[j].lower) {
1548 fprintf(stderr, "Error: invalid %s filter provided.\n", filter_term[j].name);
1549 exit(EXIT_FAILURE);
1550 }
1551 }
1552 if (filter_ptr->is_parameter) {
1553 accept = 1;
1554 filter_term = filter_ptr->filter_term;
1555 for (j = 0; j < filter_ptr->filter_terms; j++) {
1556 if (!(pardefptr = SDDS_GetParameterDefinition(&SDDS_output, filter_term[j].name)) ||
1557 (pardefptr->type == SDDS_STRING || pardefptr->type == SDDS_CHARACTER)) {
1558 fprintf(stderr, "error: unknown or non-numeric parameter %s given for filter\n", filter_term[j].name);
1559 exit(EXIT_FAILURE);
1560 }
1561 accept = SDDS_Logic(accept, SDDS_ItemInsideWindow(SDDS_GetParameter(&SDDS_output, filter_term[j].name, &result), 0, pardefptr->type, filter_term[j].lower, filter_term[j].upper), filter_term[j].logic);
1562 }
1563 if (!accept) {
1564 if (verbose)
1565 fputs(" * page failed parameter matching--continuing to next page", stderr);
1566 skip_page = 1;
1567 continue;
1568 }
1569 } else if (n_rows) {
1570 int64_t nleft;
1571 filter_term = filter_ptr->filter_term;
1572 for (j = 0; j < filter_ptr->filter_terms; j++) {
1573 if (verbose) {
1574 fprintf(stderr, " * applying filter term %" PRId64 " (column %s)", j, filter_term[j].name);
1575 fflush(stderr);
1576 }
1577 if ((nleft = SDDS_FilterRowsOfInterest(&SDDS_output, filter_term[j].name, filter_term[j].lower, filter_term[j].upper, filter_term[j].logic)) < 0) {
1578 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1579 exit(EXIT_FAILURE);
1580 }
1581 if (verbose) {
1582 fprintf(stderr, "---%" PRId64 " rows left\n", nleft);
1583 fflush(stderr);
1584 }
1585 }
1586 row_deletion = 1;
1587 }
1588 if (verbose)
1589 fprintf(stderr, "applied %s filter\n", filter_ptr->is_parameter ? "parameter" : "column");
1590 break;
1591 case IS_MATCH_DEFINITION:
1592 match_ptr = (MATCH_DEFINITION *)definition[i].structure;
1593 if (match_ptr->is_parameter) {
1594 accept = 1;
1595 match_term = match_ptr->match_term;
1596 for (j = 0; j < match_ptr->match_terms; j++) {
1597 if (!(pardefptr = SDDS_GetParameterDefinition(&SDDS_output, match_term[j].name)) ||
1598 !(pardefptr->type == SDDS_STRING || pardefptr->type == SDDS_CHARACTER)) {
1599 fprintf(stderr, "error: unknown or numeric parameter %s given for match\n", match_term[j].name);
1600 exit(EXIT_FAILURE);
1601 }
1602 if (pardefptr->type == SDDS_STRING) {
1603 char **ppc;
1604 ppc = SDDS_GetParameter(&SDDS_output, match_term[j].name, NULL);
1605 strcpy(s, *ppc);
1606 } else {
1607 char *pc;
1608 pc = SDDS_GetParameter(&SDDS_output, match_term[j].name, NULL);
1609 sprintf(s, "%c", *pc);
1610 }
1611 accept = SDDS_Logic(accept, (match_term[j].logic & SDDS_NOCASE_COMPARE ? wild_match_ci(s, match_term[j].string) : wild_match(s, match_term[j].string)), match_term[j].logic);
1612 }
1613 if (!accept) {
1614 if (verbose)
1615 fputs(" * page failed parameter matching--continuing to next page", stderr);
1616 skip_page = 1;
1617 continue;
1618 }
1619 } else if (n_rows) {
1620 match_term = match_ptr->match_term;
1621 for (j = 0; j < match_ptr->match_terms; j++)
1622 if (SDDS_MatchRowsOfInterest(&SDDS_output, match_term[j].name, match_term[j].string, match_term[j].logic) < 0) {
1623 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1624 exit(EXIT_FAILURE);
1625 }
1626 row_deletion = 1;
1627 }
1628 if (verbose)
1629 fprintf(stderr, "applied %s match\n", match_ptr->is_parameter ? "parameter" : "column");
1630 break;
1631 case IS_RPNTEST_DEFINITION:
1632 rpntest_ptr = (RPNTEST_DEFINITION *)definition[i].structure;
1633 SDDS_StoreParametersInRpnMemories(&SDDS_output);
1634 if (rpntest_ptr->is_parameter) {
1635 rpn_clear();
1636 rpn(rpntest_ptr->expression);
1637 if (rpn_check_error())
1638 exit(EXIT_FAILURE);
1639 if (!pop_log(&test_result))
1640 SDDS_Bomb("aborted due to rpn logical stack/result error for parameter-based test");
1641 rpn_clear();
1642 if (!test_result) {
1643 if (!rpntest_ptr->autostop) {
1644 if (verbose)
1645 fputs(" * page failed rpn test--continuing to next page", stderr);
1646 skip_page = 1;
1647 continue;
1648 } else {
1649 if (verbose)
1650 fputs(" * page failed rpn test--ignore remainder of file", stderr);
1651 exit(EXIT_FAILURE);
1652 }
1653 }
1654 } else if (n_rows) {
1655 if (!SDDS_FilterRowsWithRpnTest(&SDDS_output, rpntest_ptr->expression)) {
1656 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1657 exit(EXIT_FAILURE);
1658 }
1659 row_deletion = 1;
1660 }
1661 if (verbose)
1662 fprintf(stderr, "applied %s-based rpn test:\n %s\n", rpntest_ptr->is_parameter ? "parameter" : "column", rpntest_ptr->expression);
1663 break;
1664 case IS_NUMBERTEST_DEFINITION:
1665 numbertest_ptr = (NUMBERTEST_DEFINITION *)definition[i].structure;
1666 if (numbertest_ptr->is_parameter) {
1667 skip_page = !ParameterScansAsNumber(&SDDS_output, numbertest_ptr->name, numbertest_ptr->flags & NUMSCANFILTER_INVERT);
1668 } else if (n_rows) {
1669 if (SDDS_FilterRowsByNumScan(&SDDS_output, numbertest_ptr->name, numbertest_ptr->flags) == -1) {
1670 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1671 exit(EXIT_FAILURE);
1672 }
1673 row_deletion = 1;
1674 }
1675 if (verbose)
1676 fprintf(stderr, "applied %s-based %s%snumber test\n", numbertest_ptr->is_parameter ? "parameter" : "column", numbertest_ptr->flags & NUMSCANFILTER_INVERT ? " inverted " : "", numbertest_ptr->flags & NUMSCANFILTER_STRICT ? " strict " : "");
1677 break;
1678 case IS_SYSTEM_DEFINITION:
1679 system_ptr = (SYSTEM_DEFINITION *)definition[i].structure;
1680 if (system_ptr->is_parameter) {
1681 if (!system_parameter_value(&SDDS_output, system_ptr->new_name, system_ptr->source)) {
1682 fprintf(stderr, "couldn't system %s to make %s\n", system_ptr->source, system_ptr->new_name);
1683 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1684 exit(EXIT_FAILURE);
1685 }
1686 } else if (n_rows && !system_column_value(&SDDS_output, system_ptr->new_name, system_ptr->source)) {
1687 fprintf(stderr, "couldn't system %s to make %s\n", system_ptr->source, system_ptr->new_name);
1688 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1689 exit(EXIT_FAILURE);
1690 }
1691 if (verbose)
1692 fprintf(stderr, "systemed %s %s to produce %s\n", system_ptr->is_parameter ? "parameter" : "column", system_ptr->source, system_ptr->new_name);
1693 break;
1694 case IS_RPNEXPRESSION_DEFINITION:
1695 rpnexpression_ptr = (RPNEXPRESSION_DEFINITION *)definition[i].structure;
1696 if (rpnexpression_ptr->repeat == -1)
1697 continue;
1698 SDDS_StoreParametersInRpnMemories(&SDDS_output);
1699 SDDS_StoreColumnsInRpnArrays(&SDDS_output);
1700 rpn(rpnexpression_ptr->expression);
1701 if (rpn_check_error())
1702 exit(EXIT_FAILURE);
1703 rpn_clear();
1704 if (verbose)
1705 fprintf(stderr, "executed rpn expression:\n %s\n", rpnexpression_ptr->expression);
1706 if (!rpnexpression_ptr->repeat)
1707 rpnexpression_ptr->repeat = -1; /* already done, don't repeat */
1708 break;
1709 case IS_CLIP_DEFINITION:
1710 if (n_rows) {
1711 clip_ptr = (CLIP_DEFINITION *)definition[i].structure;
1712 if (!SDDS_SetRowFlags(&SDDS_output, clip_ptr->invert ? 0 : 1) ||
1713 (clip_ptr->head &&
1714 !SDDS_AssertRowFlags(&SDDS_output, SDDS_INDEX_LIMITS, (int64_t)0, (int64_t)(clip_ptr->head - 1), clip_ptr->invert ? 1 : 0)) ||
1715 (n_rows >= clip_ptr->tail && clip_ptr->tail && !SDDS_AssertRowFlags(&SDDS_output, SDDS_INDEX_LIMITS, (int64_t)(n_rows - clip_ptr->tail), (int64_t)(n_rows - 1), clip_ptr->invert ? 1 : 0))) {
1716 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1717 exit(EXIT_FAILURE);
1718 }
1719 row_deletion = 1;
1720 }
1721 break;
1722 case IS_FCLIP_DEFINITION:
1723 if (n_rows) {
1724 fclip_ptr = (FCLIP_DEFINITION *)definition[i].structure;
1725 fclip_ptr->head = fclip_ptr->fhead * n_rows;
1726 fclip_ptr->tail = fclip_ptr->ftail * n_rows;
1727 if (!SDDS_SetRowFlags(&SDDS_output, fclip_ptr->invert ? 0 : 1) ||
1728 (fclip_ptr->head &&
1729 !SDDS_AssertRowFlags(&SDDS_output, SDDS_INDEX_LIMITS, (int64_t)0, (int64_t)(fclip_ptr->head - 1), fclip_ptr->invert ? 1 : 0)) ||
1730 (n_rows >= fclip_ptr->tail && fclip_ptr->tail && !SDDS_AssertRowFlags(&SDDS_output, SDDS_INDEX_LIMITS, (int64_t)(n_rows - fclip_ptr->tail), (int64_t)(n_rows - 1), fclip_ptr->invert ? 1 : 0))) {
1731 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1732 exit(EXIT_FAILURE);
1733 }
1734 row_deletion = 1;
1735 }
1736 break;
1737 case IS_SPARSE_DEFINITION:
1738 if (n_rows) {
1739 int32_t *sparse_row_flag;
1740 sparse_ptr = (SPARSE_DEFINITION *)definition[i].structure;
1741 sparse_row_flag = tmalloc(sizeof(*sparse_row_flag) * n_rows);
1742 for (j = 0; j < n_rows; j++)
1743 sparse_row_flag[j] = 0;
1744 for (j = sparse_ptr->offset; j < n_rows; j += sparse_ptr->interval)
1745 sparse_row_flag[j] = 1;
1746 if (!SDDS_AssertRowFlags(&SDDS_output, SDDS_FLAG_ARRAY, sparse_row_flag, n_rows)) {
1747 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1748 exit(EXIT_FAILURE);
1749 }
1750 free(sparse_row_flag);
1751 row_deletion = 1;
1752 }
1753 break;
1754 case IS_SAMPLE_DEFINITION:
1755 if (n_rows) {
1756 int32_t *sample_row_flag;
1757 sample_ptr = (SAMPLE_DEFINITION *)definition[i].structure;
1758 sample_row_flag = tmalloc(sizeof(*sample_row_flag) * n_rows);
1759 for (j = 0; j < n_rows; j++)
1760 sample_row_flag[j] = random_1(1) < sample_ptr->fraction ? 1 : 0;
1761 if (!SDDS_AssertRowFlags(&SDDS_output, SDDS_FLAG_ARRAY, sample_row_flag, n_rows)) {
1762 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1763 exit(EXIT_FAILURE);
1764 }
1765 free(sample_row_flag);
1766 row_deletion = 1;
1767 }
1768 break;
1769 case IS_CAST_DEFINITION:
1770 cast_ptr = (CAST_DEFINITION *)definition[i].structure;
1771 if (cast_ptr->isParameter) {
1772 if (!cast_parameter_value(&SDDS_output, cast_ptr))
1773 exit(EXIT_FAILURE);
1774 } else if (n_rows && !cast_column_value(&SDDS_output, cast_ptr)) {
1775 exit(EXIT_FAILURE);
1776 }
1777 break;
1778 default:
1779 fprintf(stderr, "error: unknown definition type code %ld seen--this shouldn't happen!\n", definition[i].type);
1780 exit(EXIT_FAILURE);
1781 break;
1782 }
1783 if (n_rows && row_deletion) {
1784 if (i != (definitions - 1) && !SDDS_DeleteUnsetRows(&SDDS_output)) {
1785 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1786 exit(EXIT_FAILURE);
1787 }
1788 if ((n_rows = SDDS_CountRowsOfInterest(&SDDS_output)) <= 0 &&
1789 SDDS_ColumnCount(&SDDS_output) != 0) {
1790 if (!nowarnings)
1791 fprintf(stderr, "warning: no rows selected for page %" PRId32 "\n", SDDS_output.page_number);
1792 if (SDDS_NumberOfErrors())
1793 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1794 continue;
1795 }
1796 if (verbose)
1797 fprintf(stderr, " %" PRId64 " rows left\n", n_rows);
1798 }
1799 }
1800 if (skip_page)
1801 continue;
1802 if (verbose)
1803 fprintf(stderr, " %" PRId64 " rows selected for output\n", n_rows);
1804 if (!SDDS_WritePage(&SDDS_output)) {
1805 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1806 exit(EXIT_FAILURE);
1807 }
1808 }
1809
1810 if (page_number == 0) {
1811 fprintf(stderr, "Error: problem reading input file\n");
1812 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1813 exit(EXIT_FAILURE);
1814 }
1815
1816 if (!SDDS_Terminate(&SDDS_input) || !SDDS_Terminate(&SDDS_output)) {
1817 SDDS_PrintErrors(stderr, SDDS_VERBOSE_PrintErrors);
1818 exit(EXIT_FAILURE);
1819 }
1820
1821 if (tmpfile_used && !replaceFileAndBackUp(inputfile, outputfile))
1822 exit(EXIT_FAILURE);
1823 free_scanargs(&s_arg, argc);
1824
1825 if (processing_definition)
1826 free_processing_definiton(processing_definition, processing_definitions);
1827 if (equation_definition)
1828 free_equation_definition(equation_definition, equation_definitions);
1829
1830 return EXIT_SUCCESS;
1831}
1832
1833void free_processing_definiton(PROCESSING_DEFINITION **processing_definition, long n) {
1834 long i;
1835 for (i = 0; i < n; i++) {
1836 if (processing_definition[i]->parameter_name)
1837 free(processing_definition[i]->parameter_name);
1838 if (processing_definition[i]->column_name)
1839 free(processing_definition[i]->column_name);
1840 if (processing_definition[i]->description)
1841 free(processing_definition[i]->description);
1842 if (processing_definition[i]->symbol)
1843 free(processing_definition[i]->symbol);
1844 if (processing_definition[i]->functionOf)
1845 free(processing_definition[i]->functionOf);
1846 if (processing_definition[i]->weightBy)
1847 free(processing_definition[i]->weightBy);
1848 if (processing_definition[i]->match_value)
1849 free(processing_definition[i]->match_value);
1850 if (processing_definition[i]->match_column)
1851 free(processing_definition[i]->match_column);
1852 free(processing_definition[i]);
1853 }
1854 if (processing_definition)
1855 free(processing_definition);
1856}
1857
1858void free_equation_definition(EQUATION_DEFINITION **equation_definition, long n) {
1859 long i, j;
1860 for (i = 0; i < n; i++) {
1861 if (equation_definition[i]->text)
1862 free(equation_definition[i]->text);
1863 if (equation_definition[i]->name)
1864 free(equation_definition[i]->name);
1865 if (equation_definition[i]->equation)
1866 free(equation_definition[i]->equation);
1867 if (equation_definition[i]->udf_name && equation_definition[i]->udf_name != equation_definition[i]->equation)
1868 free(equation_definition[i]->udf_name);
1869 if (equation_definition[i]->editSelection)
1870 free(equation_definition[i]->editSelection);
1871 if (equation_definition[i]->select)
1872 free(equation_definition[i]->select);
1873 if (equation_definition[i]->argv) {
1874 for (j = 0; j < equation_definition[i]->argc; j++)
1875 free(equation_definition[i]->argv[j]);
1876 free(equation_definition[i]->argv);
1877 }
1878 free(equation_definition[i]);
1879 }
1880 if (equation_definition)
1881 free(equation_definition);
1882}
1883
1884typedef struct
1885{
1886 long column;
1887 char *equation;
1888} COLUMN_LIST;
1889
1890long SDDS_ComputeSetOfColumns(SDDS_DATASET *SDDS_dataset, long equ_begin, long equ_end, DEFINITION *definition) {
1891 EQUATION_DEFINITION *equation_ptr;
1892 long column;
1893 SDDS_LAYOUT *layout;
1894 double value;
1895 long equations_present;
1896 COLUMN_LIST *column_list;
1897 long column_list_ptr, max_column_list_counter;
1898 int64_t j;
1899
1900 column_list = NULL;
1901 column_list_ptr = -1;
1902 max_column_list_counter = 0;
1903 if (!SDDS_CheckDataset(SDDS_dataset, "SDDS_ComputeColumn"))
1904 return (0);
1905 layout = &SDDS_dataset->layout;
1906 column_list = NULL;
1907 column_list_ptr = -1;
1908 max_column_list_counter = 0;
1909 for (equations_present = equ_begin; equations_present < equ_end; equations_present++) {
1910 equation_ptr = (EQUATION_DEFINITION *)definition[equations_present].structure;
1911
1912 if (++column_list_ptr >= max_column_list_counter || !column_list)
1913 column_list = trealloc(column_list, sizeof(*column_list) * (max_column_list_counter += 4));
1914 column_list[column_list_ptr].column = SDDS_GetColumnIndex(SDDS_dataset, equation_ptr->name);
1915 if (equation_ptr->equation[0] == '@') {
1916 char **equation, *ptr, pfix[IFPF_BUF_SIZE];
1917 if (equation_ptr->equation[1] == '@') {
1918 /* Parameter contains algebraic notation */
1919 if (!(equation = SDDS_GetParameter(SDDS_dataset, equation_ptr->equation + 2, NULL)))
1920 SDDS_Bomb("unable to read parameter for equation definition");
1921 ptr = addOuterParentheses(*equation);
1922 if2pf(pfix, ptr, sizeof pfix);
1923 free(ptr);
1924 if (!SDDS_CopyString(equation, pfix)) {
1925 fprintf(stderr, "error: problem copying argument string\n");
1926 return 0;
1927 }
1928 } else {
1929 if (!(equation = SDDS_GetParameter(SDDS_dataset, equation_ptr->equation + 1, NULL)))
1930 SDDS_Bomb("unable to read parameter for equation definition");
1931 }
1932
1933 create_udf(equation_ptr->udf_name, *equation);
1934 free(*equation);
1935 }
1936 cp_str(&column_list[column_list_ptr].equation, equation_ptr->udf_name);
1937 if (column_list[column_list_ptr].column < 0 || column_list[column_list_ptr].column >= layout->n_columns)
1938 return (0);
1939 }
1940
1941 if (!SDDS_StoreParametersInRpnMemories(SDDS_dataset))
1942 return (0);
1943 if (!SDDS_StoreColumnsInRpnArrays(SDDS_dataset))
1944 return 0;
1945
1946 if (table_number_mem == -1) {
1947 table_number_mem = rpn_create_mem("table_number", 0);
1948 i_page_mem = rpn_create_mem("i_page", 0);
1949 n_rows_mem = rpn_create_mem("n_rows", 0);
1950 i_row_mem = rpn_create_mem("i_row", 0);
1951 }
1952
1953 rpn_store((double)SDDS_dataset->page_number, NULL, table_number_mem);
1954 rpn_store((double)SDDS_dataset->page_number, NULL, i_page_mem);
1955 rpn_store((double)SDDS_dataset->n_rows, NULL, n_rows_mem);
1956
1957 for (j = 0; j < SDDS_dataset->n_rows; j++) {
1958#if defined(DEBUG)
1959 fprintf(stderr, "Working on row %ld\n", j);
1960#endif
1961 if (!SDDS_StoreRowInRpnMemories(SDDS_dataset, j))
1962 return (0);
1963 column_list_ptr = 0;
1964 for (equations_present = equ_begin; equations_present < equ_end; equations_present++, column_list_ptr++) {
1965 column = column_list[column_list_ptr].column;
1966 rpn_clear();
1967 rpn_store((double)j, NULL, i_row_mem);
1968 value = rpn(column_list[column_list_ptr].equation);
1969 rpn_store(value, NULL, layout->column_definition[column].memory_number);
1970 if (rpn_check_error()) {
1971 SDDS_SetError("Unable to compute rpn expression--rpn error (SDDS_ComputeDefinedColumn)");
1972 return (0);
1973 }
1974#if defined(DEBUG)
1975 fprintf(stderr, "computed row value: %s = %e\n", layout->column_definition[column].name, value);
1976#endif
1977 switch (layout->column_definition[column].type) {
1978 case SDDS_CHARACTER:
1979 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (char)value, -1);
1980 break;
1981 case SDDS_SHORT:
1982 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (short)value, -1);
1983 break;
1984 case SDDS_USHORT:
1985 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (unsigned short)value, -1);
1986 break;
1987 case SDDS_LONG:
1988 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (int32_t)value, -1);
1989 break;
1990 case SDDS_ULONG:
1991 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (uint32_t)value, -1);
1992 break;
1993 case SDDS_LONG64:
1994 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (int64_t)value, -1);
1995 break;
1996 case SDDS_ULONG64:
1997 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (uint64_t)value, -1);
1998 break;
1999 case SDDS_FLOAT:
2000 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (float)value, -1);
2001 break;
2002 case SDDS_DOUBLE:
2003 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (double)value, -1);
2004 break;
2005 case SDDS_LONGDOUBLE:
2006 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (long double)value, -1);
2007 break;
2008 }
2009 }
2010 }
2011
2012 return (1);
2013}
2014
2015long SDDS_EvaluateColumn(SDDS_DATASET *SDDS_dataset, EVALUATE_DEFINITION *definition) {
2016 double value;
2017 char **equation;
2018 long type, column;
2019 int64_t j;
2020
2021 if (!SDDS_CheckDataset(SDDS_dataset, "SDDS_ComputeColumn"))
2022 return (0);
2023 if (!SDDS_StoreParametersInRpnMemories(SDDS_dataset))
2024 return (0);
2025 if (!SDDS_StoreColumnsInRpnArrays(SDDS_dataset))
2026 return 0;
2027
2028 if (table_number_mem == -1) {
2029 table_number_mem = rpn_create_mem("table_number", 0);
2030 i_page_mem = rpn_create_mem("i_page", 0);
2031 n_rows_mem = rpn_create_mem("n_rows", 0);
2032 i_row_mem = rpn_create_mem("i_row", 0);
2033 }
2034
2035 rpn_store((double)SDDS_dataset->page_number, NULL, table_number_mem);
2036 rpn_store((double)SDDS_dataset->page_number, NULL, i_page_mem);
2037 rpn_store((double)SDDS_dataset->n_rows, NULL, n_rows_mem);
2038
2039 if (!(equation = (char **)SDDS_GetColumn(SDDS_dataset, definition->source)))
2040 return 0;
2041
2042 type = SDDS_GetColumnType(SDDS_dataset, column = SDDS_GetColumnIndex(SDDS_dataset, definition->name));
2043
2044 for (j = 0; j < SDDS_dataset->n_rows; j++) {
2045 if (!SDDS_StoreRowInRpnMemories(SDDS_dataset, j))
2046 return (0);
2047 rpn_clear();
2048 rpn_store((double)j, NULL, i_row_mem);
2049 value = rpn(equation[j]);
2050 if (rpn_check_error()) {
2051 SDDS_SetError("Unable to compute rpn expression--rpn error (SDDS_EvaluateColumn)");
2052 return (0);
2053 }
2054 switch (type) {
2055 case SDDS_CHARACTER:
2056 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (char)value, -1);
2057 break;
2058 case SDDS_SHORT:
2059 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (short)value, -1);
2060 break;
2061 case SDDS_USHORT:
2062 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (unsigned short)value, -1);
2063 break;
2064 case SDDS_LONG:
2065 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (int32_t)value, -1);
2066 break;
2067 case SDDS_ULONG:
2068 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (uint32_t)value, -1);
2069 break;
2070 case SDDS_LONG64:
2071 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (int64_t)value, -1);
2072 break;
2073 case SDDS_ULONG64:
2074 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (uint64_t)value, -1);
2075 break;
2076 case SDDS_FLOAT:
2077 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (float)value, -1);
2078 break;
2079 case SDDS_DOUBLE:
2080 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (double)value, -1);
2081 break;
2082 case SDDS_LONGDOUBLE:
2083 SDDS_SetRowValues(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, j, column, (long double)value, -1);
2084 break;
2085 }
2086 }
2087
2088 return (1);
2089}
2090
2091long SDDS_EvaluateParameter(SDDS_DATASET *SDDS_dataset, EVALUATE_DEFINITION *definition) {
2092 double value;
2093 char **equation;
2094 long parameter;
2095
2096 if (!SDDS_StoreParametersInRpnMemories(SDDS_dataset))
2097 return (0);
2098
2099 if (table_number_mem == -1) {
2100 table_number_mem = rpn_create_mem("table_number", 0);
2101 i_page_mem = rpn_create_mem("i_page", 0);
2102 n_rows_mem = rpn_create_mem("n_rows", 0);
2103 i_row_mem = rpn_create_mem("i_row", 0);
2104 }
2105
2106 rpn_store((double)SDDS_dataset->page_number, NULL, table_number_mem);
2107 rpn_store((double)SDDS_dataset->page_number, NULL, i_page_mem);
2108 rpn_store((double)SDDS_dataset->n_rows, NULL, n_rows_mem);
2109
2110 if (!(equation = (char **)SDDS_GetParameter(SDDS_dataset, definition->source, NULL)))
2111 return 0;
2112
2113 rpn_clear();
2114 value = rpn(*equation);
2115 fprintf(stderr, "value = %e\n", value);
2116 free(*equation);
2117 if (rpn_check_error()) {
2118 SDDS_SetError("Unable to compute rpn expression--rpn error (SDDS_EvaluateParameter)");
2119 return (0);
2120 }
2121 switch (SDDS_GetParameterType(SDDS_dataset, parameter = SDDS_GetParameterIndex(SDDS_dataset, definition->name))) {
2122 case SDDS_CHARACTER:
2123 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (char)value, -1);
2124 break;
2125 case SDDS_SHORT:
2126 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (short)value, -1);
2127 break;
2128 case SDDS_USHORT:
2129 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (unsigned short)value, -1);
2130 break;
2131 case SDDS_LONG:
2132 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (int32_t)value, -1);
2133 break;
2134 case SDDS_ULONG:
2135 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (uint32_t)value, -1);
2136 break;
2137 case SDDS_LONG64:
2138 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (int64_t)value, -1);
2139 break;
2140 case SDDS_ULONG64:
2141 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (uint64_t)value, -1);
2142 break;
2143 case SDDS_FLOAT:
2144 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (float)value, -1);
2145 break;
2146 case SDDS_DOUBLE:
2147 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (double)value, -1);
2148 break;
2149 case SDDS_LONGDOUBLE:
2150 SDDS_SetParameters(SDDS_dataset, SDDS_SET_BY_INDEX | SDDS_PASS_BY_VALUE, parameter, (long double)value, -1);
2151 break;
2152 }
2153
2154 return (1);
2155}
2156
2157char **process_name_options(char **orig_name, long **orig_flag, long orig_names, char **delete, long deletes, char **retain,
2158 long retains, STRING_PAIR *rename, long renames, EDIT_NAME_REQUEST *edit_request, long edit_requests) {
2159 long i, j;
2160 char **new_name;
2161 char *ptr;
2162
2163 *orig_flag = tmalloc(sizeof(**orig_flag) * orig_names);
2164 for (i = 0; i < orig_names; i++)
2165 (*orig_flag)[i] = 1;
2166
2167 if (deletes) {
2168 for (i = 0; i < deletes; i++) {
2169 ptr = expand_ranges(delete[i]);
2170 /* free(delete[i]); */
2171 delete[i] = ptr;
2172 }
2173 for (j = 0; j < orig_names; j++) {
2174 for (i = 0; i < deletes; i++) {
2175 if (wild_match(orig_name[j], delete[i])) {
2176 (*orig_flag)[j] = 0;
2177 break;
2178 }
2179 }
2180 }
2181 }
2182
2183 if (retains) {
2184 for (i = 0; i < retains; i++) {
2185 ptr = expand_ranges(retain[i]);
2186 /* free(retain[i]); */
2187 retain[i] = ptr;
2188 }
2189 if (!deletes)
2190 for (j = 0; j < orig_names; j++)
2191 (*orig_flag)[j] = 0;
2192 for (j = 0; j < orig_names; j++) {
2193 if ((*orig_flag)[j])
2194 continue;
2195 for (i = 0; i < retains; i++) {
2196 if (wild_match(orig_name[j], retain[i])) {
2197 (*orig_flag)[j] = 1;
2198 break;
2199 }
2200 }
2201 }
2202 }
2203
2204 new_name = tmalloc(sizeof(*new_name) * orig_names);
2205 for (j = 0; j < orig_names; j++) {
2206 for (i = 0; i < renames; i++) {
2207 if (strcmp(rename[i][0], orig_name[j]) == 0) {
2208 SDDS_CopyString(new_name + j, rename[i][1]);
2209 break;
2210 }
2211 }
2212 if (i == renames)
2213 SDDS_CopyString(new_name + j, orig_name[j]);
2214 for (i = 0; i < edit_requests; i++) {
2215 char edit_buffer[256];
2216 ptr = expand_ranges(edit_request[i].match_string);
2217 /*free(edit_request[i].match_string); */
2218 edit_request[i].match_string = ptr;
2219 if (wild_match(new_name[j], edit_request[i].match_string)) {
2220 strcpy(edit_buffer, new_name[j]);
2221 if (!edit_string(edit_buffer, edit_request[i].edit_string))
2222 SDDS_Bomb("error editing name");
2223 /*free(new_name[j]); */
2224 SDDS_CopyString(&new_name[j], edit_buffer);
2225 }
2226 }
2227 }
2228
2229 return (new_name);
2230}
SDDS.h
SDDS (Self Describing Data Set) Data Types Definitions and Function Prototypes.
SDDS_CopyPage
int32_t SDDS_CopyPage(SDDS_DATASET *SDDS_target, SDDS_DATASET *SDDS_source)
Definition SDDS_copy.c:578
SDDS_SetRowValues
int32_t SDDS_SetRowValues(SDDS_DATASET *SDDS_dataset, int32_t mode, int64_t row,...)
Definition SDDS_dataprep.c:988
SDDS_SetParameters
int32_t SDDS_SetParameters(SDDS_DATASET *SDDS_dataset, int32_t mode,...)
Definition SDDS_dataprep.c:369
SDDS_AssertRowFlags
int32_t SDDS_AssertRowFlags(SDDS_DATASET *SDDS_dataset, uint32_t mode,...)
Sets acceptance flags for rows based on specified criteria.
Definition SDDS_extract.c:148
SDDS_GetColumn
void * SDDS_GetColumn(SDDS_DATASET *SDDS_dataset, char *column_name)
Retrieves a copy of the data for a specified column, including only rows marked as "of interest".
Definition SDDS_extract.c:611
SDDS_GetParameterAsDouble
double * SDDS_GetParameterAsDouble(SDDS_DATASET *SDDS_dataset, char *parameter_name, double *memory)
Retrieves the value of a specified parameter as a double from the current data table of an SDDS datas...
Definition SDDS_extract.c:2782
SDDS_CountRowsOfInterest
int64_t SDDS_CountRowsOfInterest(SDDS_DATASET *SDDS_dataset)
Counts the number of rows marked as "of interest" in the current data table.
Definition SDDS_extract.c:364
SDDS_FilterRowsByNumScan
int64_t SDDS_FilterRowsByNumScan(SDDS_DATASET *SDDS_dataset, char *filter_column, uint32_t mode)
Filters rows of interest in an SDDS dataset based on numeric scanning of a specified column.
Definition SDDS_extract.c:3697
SDDS_SetRowFlags
int32_t SDDS_SetRowFlags(SDDS_DATASET *SDDS_dataset, int32_t row_flag_value)
Sets the acceptance flags for all rows in the current data table of a data set.
Definition SDDS_extract.c:46
SDDS_ItemInsideWindow
int32_t SDDS_ItemInsideWindow(void *data, int64_t index, int32_t type, double lower_limit, double upper_limit)
Checks whether a data item is within a specified numeric window.
Definition SDDS_extract.c:4077
SDDS_MatchRowsOfInterest
int64_t SDDS_MatchRowsOfInterest(SDDS_DATASET *SDDS_dataset, char *selection_column, char *label_to_match, int32_t logic)
Matches and marks rows of interest in an SDDS dataset based on label matching.
Definition SDDS_extract.c:3506
SDDS_Logic
int32_t SDDS_Logic(int32_t previous, int32_t match, uint32_t logic)
Applies logical operations to determine the new state of a row flag based on previous and current mat...
Definition SDDS_extract.c:4165
SDDS_GetParameter
void * SDDS_GetParameter(SDDS_DATASET *SDDS_dataset, char *parameter_name, void *memory)
Retrieves the value of a specified parameter from the current data table of a data set.
Definition SDDS_extract.c:2481
SDDS_DeleteUnsetRows
int32_t SDDS_DeleteUnsetRows(SDDS_DATASET *SDDS_dataset)
Deletes rows from an SDDS dataset that are not marked as "of interest".
Definition SDDS_extract.c:3760
SDDS_GetDescription
int32_t SDDS_GetDescription(SDDS_DATASET *SDDS_dataset, char **text, char **contents)
Retrieves the text and contents descriptions from an SDDS dataset.
Definition SDDS_extract.c:4626
SDDS_FilterRowsOfInterest
int64_t SDDS_FilterRowsOfInterest(SDDS_DATASET *SDDS_dataset, char *filter_column, double lower_limit, double upper_limit, int32_t logic)
Filters rows of interest in an SDDS dataset based on numeric ranges in a specified column.
Definition SDDS_extract.c:3628
SDDS_ChangeParameterInformation
int32_t SDDS_ChangeParameterInformation(SDDS_DATASET *SDDS_dataset, char *field_name, void *memory, int32_t mode,...)
Modifies a specific field in a parameter definition within the SDDS dataset.
Definition SDDS_info.c:485
SDDS_ChangeColumnInformation
int32_t SDDS_ChangeColumnInformation(SDDS_DATASET *SDDS_dataset, char *field_name, void *memory, int32_t mode,...)
Modifies a specific field in a column definition within the SDDS dataset.
Definition SDDS_info.c:364
SDDS_InitializeInput
int32_t SDDS_InitializeInput(SDDS_DATASET *SDDS_dataset, char *filename)
Definition SDDS_input.c:49
SDDS_Terminate
int32_t SDDS_Terminate(SDDS_DATASET *SDDS_dataset)
Definition SDDS_input.c:1448
SDDS_ReadPage
int32_t SDDS_ReadPage(SDDS_DATASET *SDDS_dataset)
Definition SDDS_input.c:1006
SDDS_InitializeOutput
int32_t SDDS_InitializeOutput(SDDS_DATASET *SDDS_dataset, int32_t data_mode, int32_t lines_per_row, const char *description, const char *contents, const char *filename)
Initializes the SDDS output dataset.
Definition SDDS_output.c:601
SDDS_WritePage
int32_t SDDS_WritePage(SDDS_DATASET *SDDS_dataset)
Writes the current data table to the output file.
Definition SDDS_output.c:1228
SDDS_WriteLayout
int32_t SDDS_WriteLayout(SDDS_DATASET *SDDS_dataset)
Writes the SDDS layout header to the output file.
Definition SDDS_output.c:899
SDDS_ProcessParameterString
int32_t SDDS_ProcessParameterString(SDDS_DATASET *SDDS_dataset, char *string, int32_t mode)
Process a parameter definition string.
Definition SDDS_process.c:323
SDDS_ProcessColumnString
int32_t SDDS_ProcessColumnString(SDDS_DATASET *SDDS_dataset, char *string, int32_t mode)
Process a column definition string.
Definition SDDS_process.c:283
SDDS_TransferColumnDefinition
int32_t SDDS_TransferColumnDefinition(SDDS_DATASET *target, SDDS_DATASET *source, char *name, char *newName)
Transfers a column definition from a source dataset to a target dataset.
Definition SDDS_transfer.c:35
SDDS_TransferArrayDefinition
int32_t SDDS_TransferArrayDefinition(SDDS_DATASET *target, SDDS_DATASET *source, char *name, char *newName)
Transfers an array definition from a source dataset to a target dataset.
Definition SDDS_transfer.c:111
SDDS_TransferParameterDefinition
int32_t SDDS_TransferParameterDefinition(SDDS_DATASET *target, SDDS_DATASET *source, char *name, char *newName)
Transfers a parameter definition from a source dataset to a target dataset.
Definition SDDS_transfer.c:73
SDDS_SetError
void SDDS_SetError(char *error_text)
Records an error message in the SDDS error stack.
Definition SDDS_utils.c:379
SDDS_GetParameterType
int32_t SDDS_GetParameterType(SDDS_DATASET *SDDS_dataset, int32_t index)
Retrieves the data type of a parameter in the SDDS dataset by its index.
Definition SDDS_utils.c:2251
SDDS_GetParameterDefinition
PARAMETER_DEFINITION * SDDS_GetParameterDefinition(SDDS_DATASET *SDDS_dataset, char *name)
Retrieves the definition of a specified parameter from the SDDS dataset.
Definition SDDS_utils.c:1075
SDDS_GetParameterNames
char ** SDDS_GetParameterNames(SDDS_DATASET *SDDS_dataset, int32_t *number)
Retrieves the names of all parameters in the SDDS dataset.
Definition SDDS_utils.c:2501
SDDS_GetParameterIndex
int32_t SDDS_GetParameterIndex(SDDS_DATASET *SDDS_dataset, char *name)
Retrieves the index of a named parameter in the SDDS dataset.
Definition SDDS_utils.c:1338
SDDS_GetColumnIndex
int32_t SDDS_GetColumnIndex(SDDS_DATASET *SDDS_dataset, char *name)
Retrieves the index of a named column in the SDDS dataset.
Definition SDDS_utils.c:1309
SDDS_ColumnCount
int32_t SDDS_ColumnCount(SDDS_DATASET *page)
Retrieves the number of columns in the SDDS dataset.
Definition SDDS_utils.c:5007
SDDS_CheckColumn
int32_t SDDS_CheckColumn(SDDS_DATASET *SDDS_dataset, char *name, char *units, int32_t type, FILE *fp_message)
Checks if a column exists in the SDDS dataset with the specified name, units, and type.
Definition SDDS_utils.c:4562
SDDS_GetColumnNames
char ** SDDS_GetColumnNames(SDDS_DATASET *SDDS_dataset, int32_t *number)
Retrieves the names of all columns in the SDDS dataset.
Definition SDDS_utils.c:2460
SDDS_CheckDataset
int32_t SDDS_CheckDataset(SDDS_DATASET *SDDS_dataset, const char *caller)
Validates the SDDS dataset pointer.
Definition SDDS_utils.c:552
SDDS_PrintErrors
void SDDS_PrintErrors(FILE *fp, int32_t mode)
Prints recorded error messages to a specified file stream.
Definition SDDS_utils.c:432
SDDS_DeleteParameterFixedValues
int32_t SDDS_DeleteParameterFixedValues(SDDS_DATASET *SDDS_dataset)
Deletes fixed values from all parameters in the SDDS dataset.
Definition SDDS_utils.c:4883
SDDS_ClearErrors
void SDDS_ClearErrors()
Clears all recorded error messages from the SDDS error stack.
Definition SDDS_utils.c:318
SDDS_ApplyFactorToColumn
int32_t SDDS_ApplyFactorToColumn(SDDS_DATASET *SDDS_dataset, char *name, double factor)
Applies a scaling factor to all elements of a specific column in the SDDS dataset.
Definition SDDS_utils.c:3123
SDDS_RegisterProgramName
void SDDS_RegisterProgramName(const char *name)
Registers the executable program name for use in error messages.
Definition SDDS_utils.c:288
SDDS_NumberOfErrors
int32_t SDDS_NumberOfErrors()
Retrieves the number of errors recorded by SDDS library routines.
Definition SDDS_utils.c:304
SDDS_ApplyFactorToParameter
int32_t SDDS_ApplyFactorToParameter(SDDS_DATASET *SDDS_dataset, char *name, double factor)
Applies a scaling factor to a specific parameter in the SDDS dataset.
Definition SDDS_utils.c:3046
SDDS_GetColumnType
int32_t SDDS_GetColumnType(SDDS_DATASET *SDDS_dataset, int32_t index)
Retrieves the data type of a column in the SDDS dataset by its index.
Definition SDDS_utils.c:2153
SDDS_Bomb
void SDDS_Bomb(char *message)
Terminates the program after printing an error message and recorded errors.
Definition SDDS_utils.c:342
SDDS_CheckParameter
int32_t SDDS_CheckParameter(SDDS_DATASET *SDDS_dataset, char *name, char *units, int32_t type, FILE *fp_message)
Checks if a parameter exists in the SDDS dataset with the specified name, units, and type.
Definition SDDS_utils.c:4653
SDDS_GetArrayNames
char ** SDDS_GetArrayNames(SDDS_DATASET *SDDS_dataset, int32_t *number)
Retrieves the names of all arrays in the SDDS dataset.
Definition SDDS_utils.c:2539
SDDS_CopyString
int32_t SDDS_CopyString(char **target, const char *source)
Copies a source string to a target string with memory allocation.
Definition SDDS_utils.c:856
SDDS_PrintTypedValue
int32_t SDDS_PrintTypedValue(void *data, int64_t index, int32_t type, char *format, FILE *fp, uint32_t mode)
Prints a data value of a specified type using an optional printf format string.
Definition SDDS_utils.c:59
SDDSaps.h
Header file for routines used by SDDS command-line applications.
edit_string
long edit_string(char *text, char *edit)
Edits the provided text based on the specified edit commands.
Definition edit_string.c:75
SDDS_ULONG
#define SDDS_ULONG
Identifier for the unsigned 32-bit integer data type.
Definition SDDStypes.h:67
SDDS_FLOAT
#define SDDS_FLOAT
Identifier for the float data type.
Definition SDDStypes.h:43
SDDS_STRING
#define SDDS_STRING
Identifier for the string data type.
Definition SDDStypes.h:85
SDDS_ULONG64
#define SDDS_ULONG64
Identifier for the unsigned 64-bit integer data type.
Definition SDDStypes.h:55
SDDS_LONG
#define SDDS_LONG
Identifier for the signed 32-bit integer data type.
Definition SDDStypes.h:61
SDDS_SHORT
#define SDDS_SHORT
Identifier for the signed short integer data type.
Definition SDDStypes.h:73
SDDS_CHARACTER
#define SDDS_CHARACTER
Identifier for the character data type.
Definition SDDStypes.h:91
SDDS_USHORT
#define SDDS_USHORT
Identifier for the unsigned short integer data type.
Definition SDDStypes.h:79
SDDS_DOUBLE
#define SDDS_DOUBLE
Identifier for the double data type.
Definition SDDStypes.h:37
SDDS_LONGDOUBLE
#define SDDS_LONGDOUBLE
Identifier for the long double data type.
Definition SDDStypes.h:31
SDDS_LONG64
#define SDDS_LONG64
Identifier for the signed 64-bit integer data type.
Definition SDDStypes.h:49
trealloc
void * trealloc(void *old_ptr, uint64_t size_of_block)
Reallocates a memory block to a new size.
Definition array.c:181
tmalloc
void * tmalloc(uint64_t size_of_block)
Allocates a memory block of the specified size with zero initialization.
Definition array.c:59
cp_str
char * cp_str(char **s, char *t)
Copies a string, allocating memory for storage.
Definition cp_str.c:28
delete_chars
char * delete_chars(char *s, char *t)
Removes all occurrences of characters found in string t from string s.
Definition delete_chars.c:28
random_1
double random_1(long iseed)
Generate a uniform random double in [0,1] using a custom seed initialization.
Definition drand.c:175
fexists
long fexists(const char *filename)
Checks if a file exists.
Definition fexists.c:27
match_string
long match_string(char *string, char **option, long n_options, long mode)
Matches a given string against an array of option strings based on specified modes.
Definition match_string.c:34
replaceFileAndBackUp
long replaceFileAndBackUp(char *file, char *replacement)
Replaces a file with a replacement file and creates a backup of the original.
Definition replacefile.c:75
scanargs
int scanargs(SCANNED_ARG **scanned, int argc, char **argv)
Definition scanargs.c:36
processPipeOption
long processPipeOption(char **item, long items, unsigned long *flags)
Definition scanargs.c:356
processFilenames
void processFilenames(char *programName, char **input, char **output, unsigned long pipeFlags, long noWarnings, long *tmpOutputUsed)
Definition scanargs.c:390
free_scanargs
void free_scanargs(SCANNED_ARG **scanned, int argc)
Definition scanargs.c:584
scanItemList
long scanItemList(unsigned long *flags, char **item, long *items, unsigned long mode,...)
Scans a list of items and assigns values based on provided keywords and types.
Definition scanitemlist.c:41
CAST_DEFINITION
Definition SDDSaps.h:201
CLIP_DEFINITION
Definition SDDSaps.h:172
COLUMN_LIST
Definition sddslocaldensity.c:156
CONVERSION_DEFINITION
Definition SDDSaps.h:124
EDIT_DEFINITION
Definition SDDSaps.h:73
EDIT_NAME_REQUEST
Definition sddsconvert.c:162
EQUATION_DEFINITION
Definition SDDSaps.h:58
EVALUATE_DEFINITION
Definition SDDSaps.h:227
FCLIP_DEFINITION
Definition SDDSaps.h:208
FILTER_DEFINITION
Definition SDDSaps.h:142
FILTER_TERM
Definition SDDSaps.h:136
FORMAT_DEFINITION
Definition SDDSaps.h:195
IFITEM_LIST
Definition SDDSaps.h:49
MATCH_DEFINITION
Definition SDDSaps.h:148
MATCH_TERM
Definition SDDSaps.h:131
NUMBERTEST_DEFINITION
Definition SDDSaps.h:188
PARAMETER_DEFINITION
Definition SDDS.h:171
PRINT_DEFINITION
Definition SDDSaps.h:81
PROCESSING_DEFINITION
Definition SDDSaps.h:88
RPNEXPRESSION_DEFINITION
Definition SDDSaps.h:166
RPNTEST_DEFINITION
Definition SDDSaps.h:154
SAMPLE_DEFINITION
Definition SDDSaps.h:183
SCAN_DEFINITION
Definition SDDSaps.h:67
SDDS_DATASET
Definition SDDS.h:308
SDDS_LAYOUT
Definition SDDS.h:257
SPARSE_DEFINITION
Definition SDDSaps.h:178
SYSTEM_DEFINITION
Definition SDDSaps.h:160
TIME_FILTER_DEFINITION
Definition SDDSaps.h:215
wild_match_ci
int wild_match_ci(char *string, char *template)
Determine whether one string is a case-insensitive wildcard match for another.
Definition wild_match.c:220
expand_ranges
char * expand_ranges(char *template)
Expand range specifiers in a wildcard template into explicit character lists.
Definition wild_match.c:429
wild_match
int wild_match(char *string, char *template)
Determine whether one string is a wildcard match for another.
Definition wild_match.c:49
Generated by doxygen 1.12.0