Data Files
SpecForge supports three data formats for signal data: CSV, JSON, and JSONL (also known as NDJSON). Parameters are provided separately in a JSON file.
Signal data
Each data sample has a time field and one or more signal values.
The time field is required and must be named exactly time (case-sensitive).
It accepts any numeric value (integer or floating point) and samples should be in increasing time order.
CSV
Each column header maps to a signal name.
Record-typed signals are flattened with a separator (default _).
time,speed,ambient_temp,gps_lat,gps_lon
0.0,0.0,25.0,34.634,135.996
1.0,30.0,26.0,34.635,135.997
Booleans in CSV can be written as true/false, True/False, or 1/0.
Enum constructors are written as they are in source files.
But because there is no notion of modules and imports in data files, the constructors can only contain the name, without module and type qualifications
(e.g., #Color::Red and #Utils::Color::Red are not allowed, they should be written as #Red).
Constructors in data files are be properly disambiguated.
JSON
A JSON array of objects, each with time and value keys.
Record-typed signals can use nested JSON objects:
[
{
"time": 0.0,
"value": {
"speed": 0.0,
"ambient_temp": 25.0,
"gps": { "lat": 34.634, "lon": 135.996 }
}
},
{
"time": 1.0,
"value": {
"speed": 30.0,
"ambient_temp": 26.0,
"gps": { "lat": 34.635, "lon": 135.997 }
}
}
]
They can also be flattened (see Record encoding):
[
{
"time": 0.0,
"value": {
"speed": 0.0,
"ambient_temp": 25.0,
"gps_lat": 34.634,
"gps_lon": 135.996
}
},
{
"time": 1.0,
"value": {
"speed": 30.0,
"ambient_temp": 26.0,
"gps_lat": 34.635,
"gps_lon": 135.997
}
}
]
JSONL
Same as JSON, but one object per line without the outer array.
{"time": 0.0, "value": {"speed": 0.0, "ambient_temp": 25.0, "gps": {"lat": 34.634, "lon": 135.996}}}
{"time": 1.0, "value": {"speed": 30.0, "ambient_temp": 26.0, "gps": {"lat": 34.635, "lon": 135.997}}}
Column names
The column names in a data file must match the signal declarations in the spec (or one of their aliases). How signal names map to column names depends on two things: the component hierarchy and the record encoding.
Note: Extra columns in data files that do not match any signal declaration are silently ignored.
Component hierarchy
Signals from system components are prefixed with the component path, separated by ::.
For instance, given a Battery system:
system Battery
signal level: Float
signal voltage: Float
And a Vehicle that uses it:
system Vehicle
signal speed: Float
component battery: Battery
The data file needs columns for speed (a direct signal) and battery::level, battery::voltage (signals from the Battery component).
In CSV this looks like:
time,speed,battery::level,battery::voltage
0.0,0.0,80.0,7.4
In JSON, components can also be written as nested objects:
{
"time": 0.0,
"value": {
"speed": 0.0,
"battery": { "level": 80.0, "voltage": 7.4 }
}
}
Record encoding
Signals with record types need their fields represented as separate columns. There are two encoding modes.
Flat (default): record fields are concatenated with a separator (default _).
This applies to all formats.
For example, given:
signal gps: { lat: Float, lon: Float }
The columns are gps_lat and gps_lon.
time,gps_lat,gps_lon
0.0,34.634,135.996
A custom separator can be specified. Forbidden characters differ by format:
- CSV: the separator cannot contain
,(the CSV delimiter) or::(the qualified-name separator). - JSON/JSONL: the separator cannot contain
"(string-key delimiter),\(escape character), or::(the qualified-name separator).
Nested: record fields are represented as nested JSON objects. This only applies to JSON and JSONL.
{
"time": 0.0,
"value": { "gps": { "lat": 34.634, "lon": 135.996 } }
}
Aliases
If a signal or parameter has an alias, that alias may be used as the column (or field) name in a data file instead of the name declared in the source file. Column (or field) names may contain any combination of aliases (regardless of language) and canonical names, as long as they are unambiguous.
For example, for the following system
system Vehicle
#[alias(en = "Velocity", ja = "速度")]
signal speed: Float
#[alias(en = "Pressure", ja = "圧力")]
signal pressure: Float
#[alias(en = "Temperature", ja = "温度")]
signal temp: Float
one can mix and match canonical names and aliases in a CSV file:
time,Velocity,圧力,temp
0.0,30.0,101.325,295.0
For record-typed signals and params, the alias replaces the signal’s root name.
Given #[alias(en = "Position")] signal gps: { lat: Float, lon: Float }, the flattened columns become Position_lat and Position_lon (flat encoding), or the object key Position (nested encoding).
The record’s field names (lat, lon) are unaffected.
Mapped signals
When instantiating a component, any of its signals (or parameters) can be mapped to an expression. A mapped signal is computed from other signals in the parent system rather than read from data, so it does not need a column in the data file.
For example, given a PowerUnit with two signals:
system PowerUnit
param max_output: Float
signal output: Float
signal temperature: Float
A Vehicle can map temperature to an expression:
system Vehicle
signal speed: Float
signal ambient_temp: Float
component power: PowerUnit {
param max_output = 100.0
signal temperature = ambient_temp + speed * 0.5
}
Here power::temperature is computed from the parent signals ambient_temp and speed, so it does not need a column in the data file.
Similarly, power::max_output is mapped to 100.0 and does not need to appear in the parameter file.
Only unmapped signals like power::output require data columns.
Parameter files
The structure of a parameter file mirrors the param declarations in the spec.
The Vehicle system declares its own parameter and provides a value for PowerUnit’s:
system Vehicle
param temp_threshold: Float
component power: PowerUnit {
param max_output = 100.0
}
A corresponding parameter file:
{
"temp_threshold": 50.0
}
Parameters with default values can be omitted.
In the example above, power::max_output has a default of 100.0 and is not present in the file.
Putting it all together
Assembling the Battery and PowerUnit systems from above:
system Vehicle
signal speed: Float
signal ambient_temp: Float
signal `fuel level`: Float
signal gps: { lat: Float, lon: Float }
param temp_threshold: Float
component battery: Battery
component power: PowerUnit {
param max_output = 100.0
signal temperature = ambient_temp + speed * 0.5
}
The parameter file provides temp_threshold and omits power::max_output (it has a default):
{
"temp_threshold": 50.0
}
A CSV data file for this system (flat encoding, _ separator):
time,speed,ambient_temp,fuel level,gps_lat,gps_lon,battery::level,battery::voltage,power::output
0.0,0.0,25.0,100.0,34.634,135.996,80.0,7.4,0.0
1.0,30.0,26.0,99.8,34.635,135.997,78.0,7.3,40.0
Note that `fuel level` appears as fuel level without backticks, and power::temperature is absent because it is a mapped signal.
The same data in JSON (nested encoding):
[
{
"time": 0.0,
"value": {
"speed": 0.0,
"ambient_temp": 25.0,
"fuel level": 100.0,
"gps": { "lat": 34.634, "lon": 135.996 },
"battery": { "level": 80.0, "voltage": 7.4 },
"power": { "output": 0.0 }
}
},
{
"time": 1.0,
"value": {
"speed": 30.0,
"ambient_temp": 26.0,
"fuel level": 99.8,
"gps": { "lat": 34.635, "lon": 135.997 },
"battery": { "level": 78.0, "voltage": 7.3 },
"power": { "output": 40.0 }
}
}
]
The schema command
When dealing with complex system consisting of a deep hierarchy of components, ensuring that signal or parameter files are in the correct format can become tedious. The schema command can be used to list the expected signals/parameters, bootstap data files, and check existing files against a system’s schema.
Notes
- Integer values are automatically coerced to Float when the signal type is
Float. - The data format is auto-detected from the file extension (
.csv,.json,.jsonl) but can be overridden with the--file-formatCLI flag.