Workflows and Workflow Patterns

The Orchestration System organizes Tasks, Commands, and Actions in the form of Workflows. Dependencies exist within each workflow between Tasks, Commands, and Actions, while individual Workflows are independent from each other. The Orchestration System may place and schedule the execution of the Tasks, Commands, and Actions of more than one Workflow at the same time. A Workflow may follow a Workflow Pattern, which is a solution template to a repeatedly ocurring Workflow problem.

Workflows

The term workflow is quite overloaded in the computing community. In the INTERSECT federated ecosystem, a workflow contains one or more tasks, where each task contains one or more commands, and each command contains one or more actions (see Tasks, Commands, and Actions). There is a defined dependency for each task within a workflow, for each command within a task, and for each action within a command. Dependencies can be serial (one after another), parallel (simultaneously), or a mixture of both, but are directed (clear happens before relationships). The directed graph created by these dependencies can be acyclic (Fig. 27) or cyclic (Fig. 28). There is, however, a clear start and a clear end, where ending a cyclic directed graph requires a condition in one of the involved tasks/commands/actions involved in the cycle to end it.

Example of an acyclic workflow

Fig. 27 Example of an acyclic INTERSECT workflow using tasks containing one or more commands, and each command containing one or more actions

Example of a cyclic workflow

Fig. 28 Example of a cyclic INTERSECT workflow using tasks containing one or more commands, and each command containing one or more actions

Workflow Patterns

A workflow pattern is a design pattern that describes a generalized solution to a recurring workflow problem within a well-defined context. As with any design pattern, a workflow pattern is often created from best practices and contains the essential elements of the problems it tackles and the corresponding solution(s). It offers a template on how to solve a specific workflow problem that may apply to different situations. It may also describe different solution alternatives to a specific problem. Different types of workflow patterns exist that address different problem categories:

Workflow Execution Patterns

Workflow execution patterns describe recurring problems and corresponding solutions in detailed workflow functionality [B26], such as execution graphs, decision points, and synchronization points. These execution patterns can be classified into different categories:

  • Basic control (e.g., sequence, parallel split, exclusive choice, and simple merge)

  • Advanced branching and synchronization (e.g., multiple choice, conditional choice, and multiple merge)

  • Structural (e.g., arbitrary cycle and implicitly terminate)

  • Multiple instances (e.g., without synchronizing and with a prior known design time knowledge)

  • State-based (e.g., deferred choice, interleaved parallel routing, and milestone)

  • Cancellation (e.g., cancel activity, cancel case, and cancel wait)

Workflow Motifs

Workflow motifs describe recurring problems and corresponding solutions in workflow task organization at a higher abstraction level, such as data movement and data analysis in computational scientific workflows [B27]. These workflow motifs can be classified into different categories:

  • Data-oriented (e.g., data preparation, cleaning/curation, movement, retrieval, visualization, and publication)

  • Inter workflow-oriented (e.g., atomic, composite, and overloading)

  • Intra workflow-oriented (e.g., macros, human interactions, and stateful/asynchronous invocations)

Computational/Experimental Science Workflow Patterns

Computational/experimental science workflow patterns describe recurring problems and corresponding solutions in workflow task organization for a specific domain, such as coupled computational simulations, data collection/analysis for experiment steering, or model training for design of experiments [B28]. The INTERSECT Science Use Case Design Patterns fit under this description, so do the Integrated Research Infrastructure Architecture Patterns.

The INTERSECT Science Use Case Design Patterns follow a formal description format that is common to design patterns (e.g., name, context, problem, forces, solution, resulting context, related patterns, examples, and known uses). The Integrated Research Infrastructure Architecture Patterns and the patterns for scientific instruments and computation in [B28] are more informal descriptions, similar to workflow motifs.