This vignette describes version 1.0 of the ROCK project file format.
ROCK project files have extension .ROCKproject and are
ZIP archives. They contain two things:
Files containing the data, ideally in a deliberately designed set of sub-directories to facilitate tracing the data through different stages of processing and analysis;
Files containing settings and directives for applications and processing of the data.
The former are raw data files and ROCK files. ROCK files are plain
text files with the .rock extension.
The latter are YAML files. Of these, the only required one is the
_ROCKproject.yml file. This file must always be a regular
YAML file that contains a map with key _ROCKproject. This
map in turn must contain maps with keys project,
codebook, sources, and
workflow.
The project map contains project metadata, such as the
project’s title, its authors, optional (but
strongly recommended!) author identifiers in authorIds, the
project’s version, the version of the ROCK standard used in
the project (with key ROCK_version), the version of the
ROCK project file (with key ROCK_project_version), the date
the project was created (with key date_created), and the
date the project was last modified (with key
date_modified).
The codebook map contains the project’s codebook, either
embedded or by linking to it. The codebook key can also
have value ~ (NULL) if not codebook information is
specified (or the codebook is embedded in the ROCK files). Valid keys to
be specified with the codebook map are urcid,
embedded, and local. The urcid
key can store the project’s Unique ROCK Codebook Identifier (i.e. its
URCID) as a URL to a ROCK codebook in spreadsheet (.xlsx or
.ods format) or YAML (.yml or
.rock) format.
The sources map specifies where the project’s data
resides. This is specified in terms of regular expressions. The first
valid key is extension, which is not a regular expression
but can be used to conveniently specify that files with a given
extension must be imported. This is used if regex is
~ (NULL, i.e. unspecified). However, if a value is
specified for regex, a program importing a ROCK project
should ignore whatever is specified for extension. The
value stored in the dirsToIncludeRegex key should be a
regular expression indicating which directories contain the data
(i.e. the ROCK files forming the project). The recursive
key can be true or false and indicates whether
all subdirectories of matched directories should be imported too. The
dirsToExcludeRegex regular expression can be used to ignore
directories. In addition, if filesToIncludeRegex is
specified, only files matching that regular expression should be
imported; and if filesToExcludeRegex is specified, files
matching that regular expression should be ignored.
Finally, the workflow map described the workflow and
data management template used in this project. It consists of a
pipeline and actions. The
pipeline is a sequence of stages, each with an identifier
(in key stage); the directory containing files in that
stage (in key dirName; note that this is a single directory
name, not a regular expression!); and a sequence of one or more next
stage (with key nextStages). Each element in
nextStages has a nextStageId key and a
actionId. The nextStageId specifies to which
stage files transfer (i.e. are saved) when the action with the
corresponding actionId is executed. These
actions are stored in a sequence where each element has an
actionId; a language specified the programming
language the action is specified in; one or more
dependencies (typically packages that need to be loaded in
that programming environment before the script can be
executed), and a script section specifying the commands to
run to execute that action. In this script, two placeholders can be
used: {currentStage::dirName} will be replaced with the
contents of dirName for the current stage; and
{nextStage::dirName} will be replaced with the contents of
dirName for the next stage. The latter part of these
expressions (dirName in both of these examples) can be
replaced by other keys specified in each stage to allow setting
parameters in the pipeline specification.
An example of a _ROCKproject.yml file is included
below.
_ROCKproject:
project:
title: "The Alice Study" # Any character string
authors: "Author names as string" # Any character string
authorIds:
-
display_name: "Talea Cornelius" # Any character string
orcid: "0000-0001-7181-0981" # Any character string matching ^([0-9]{4}-){3}[0-9]{4}$
shorcid: "ip6b381" # Any character string matching ^([0-9a-zA-Z]+$
-
display_name: "Gjalt-Jorn Peters" # Any character string
orcid: "0000-0002-0336-9589" # Any character string matching ^([0-9]{4}-){3}[0-9]{4}$
shorcid: "it36ll9" # Any character string matching ^([0-9a-zA-Z]+$
version: "1.1" # Anything matching regex [0-9]+(\\.[0-9]+)*
ROCK_version: 1 # Anything matching regex [0-9]+(\\.[0-9]+)*
ROCK_project_version: 1 # Anything matching regex [0-9]+(\\.[0-9]+)*
date_created: "2023-03-01 20:03:51 UTC" # Anything matching that date format, preferably converted to UTC timezone
date_modified: "2023-03-08 20:03:51 UTC" # Anything matching that date format, preferably converted to UTC timezone
codebook:
urcid: ""
embedded: ~
local: ""
sources:
extension: ".rock" # Any valid extension
regex: ~ # Any regex or ~
dirsToIncludeRegex: data/ # Any regex or ~
recursive: true # true or false
dirsToExcludeRegex: ~ # Any regex or ~
filesToIncludeRegex: ~ # Any regex or ~
filesToExcludeRegex: ~ # Any regex or ~
workflow:
pipeline:
-
stage: raw # Anything matching regex [a-A-Z][a-zA-Z0-9_]*
dirName: "data/010---raw-sources" # Any valid directory name, using a forward slash as separator
nextStages:
-
nextStageid: clean # A different stage identifier or ~
actionId: cleanSource
-
nextStageid: uids # A different stage identifier or ~
actionId: addUIDs
-
stage: clean # Anything matching regex [a-A-Z][a-zA-Z0-9_]*
dirName: "data/020---cleaned-sources" # Any valid directory name, using a forward slash as separator
nextStages:
-
nextStageid: uids # A different stage identifier or ~
actionId: addUIDs
-
stage: uids # Anything matching regex [a-A-Z][a-zA-Z0-9_]*
dirName: "data/030---sources-with-uids" # Any valid directory name, using a forward slash as separator
nextStage: coded # A different stage identifier or ~
-
stage: coded # Anything matching regex [a-A-Z][a-zA-Z0-9_]*
dirName: "data/040---coded-sources" # Any valid directory name, using a forward slash as separator
nextStage: masked # A different stage identifier or ~
-
stage: masked # Anything matching regex [a-A-Z][a-zA-Z0-9_]*
dirName: "data/090---masked-sources" # Any valid directory name, using a forward slash as separator
nextStage: ~ # A different stage identifier or ~
actions:
-
actionId: addUIDs # String, referenced from the stages
language: R # Language, has to be matched to interpreter
dependencies: rock # Dependencies to be loaded before running the script
script: | # Literal block style string
rock::prepend_ids_to_sources(
input = {currentStage::dirName},
output = {nextStage::dirName}
);