Reinitialization

The Reinitialization structural pattern is a derivative of the Reconfiguration architectural pattern and the Fault Treatment and Fault Recovery strategy patterns in the original resilience design pattern specification (Fig. 32) [B24]. It offers detection, containment, and mitigation, but does not necessarily restore the preserved state of the entire system. The following describes the Pattern and its application in the System Scope and in the Service Scope of the INTERSECT federated ecosystem for instrument science. Note that the Pattern description uses the terms system, subsystem, and service in an abstract way, while the System Scope and the Service Scope map those terms to the INTERSECT federated ecosystem.

Pattern

Problem

A hardware or software error or subsystem failure due to a physical fault (e.g., wear-out or destruction) or a design fault (e.g., resource underprovisioning) causes a software, such as a service, to experience an error or failure.

Context

The pattern applies to a system that has the following characteristics:

  • The system is deterministic, i.e., forward progress of the system is defined in terms of the input state to the system and the execution steps completed since system initialization.

  • The system has well-defined parameters that enable a monitoring system to discover the presence of an existing or future fault, error, or failure in the behavior of the monitored system.

  • The interaction between the monitored and monitoring systems is bounded in terms of time.

  • The monitoring system has the capability to readily analyze the behavior of the monitored system to detect or predict a fault, error, or failure.

  • The experienced fault, error, or failure must not be persistent.

Forces

  • The interactions between the monitoring and monitored systems may interfere with the operation of the monitored system, specifically its performance.

  • While the frequency and duration of these interactions must be minimized to reduce the monitoring overhead, the interactions must be able to detect or predict a fault, error, or failure in the monitored system.

  • The diagnostic resolution, i.e., the degree of accuracy of the fault diagnosis, must be high to be effective.

  • The ability of the system to reinitialize subsystems or the entire system must permit system operation that is functionally equivalent to the fault-, error-, and failure-free operation.

  • The reinitializing of a subsystem or system is often a slow process and causes an interruption.

Solution

The pattern alleviates the impact of a fault, error, or failure on system operation by restoring the affected subsystem or system to its initial state. It has a detection and a containment and mitigation component that acts upon the notification from the detection component.

The detection component enables the discovery of an existing or future fault, error, or failure in the system. It enables its containment and mitigation by notifying the system about it with location and type information. The solution requires a monitoring system, which may be a subsystem of the monitored system or an external independent system, to observe key parameters of the monitored system and to notify the monitored system when these parameters deviate. This pattern does not specify the detailed method of detection or prediction.

The containment and mitigation component acts upon the notification from the detection component by restoring the affected subsystem or system to its initial state, i.e., a reset. The pattern may create subsystems out of the system to temporarily isolate, but not permanently exclude, the affected subsystem. The interconnection between the subsystems is restored after subsystem or system reinitialization. The components of this pattern are illustrated in Fig. 45.

Reinitialization pattern components

Fig. 45 Reinitialization pattern components

Capability

A system using this pattern is able to continue to operate in the presence of a non-permanent fault, error, or failure with significant interruption and all or at least some loss of progress. This pattern provides detection/prediction, containment, and mitigation of a future or existing fault, error, or failure in the system by restoring the affected subsystem or system to its initial state. The flowchart of the pattern is shown in Fig. 46, the state diagram in Fig. 47, and its parameters in Table 8.

Flowchart

Fig. 46 Flowchart

State diagram

Fig. 47 State diagram

Table 8 Reinitialization pattern parameters

Parameter

Definition

\(T_{e}\)

Time to execute system progress

\(T_{d}\)

Time to detect or predict a fault, error or failure

\(T_{i}\)

Time to isolate the affected subsystem(s)

\(T_{r}\)

Time to reset the entire system or affected subsystem(s)
Protection Domain

The protection domain extends to the monitored system’s state that is not lost due to a subsystem or system reinitialization and to all of its resources that are able to be operate correctly after a reinitialization.

Resulting Context

A fault, error, or failure is prevented from affecting the correct operation of the system. While functional equivalency to the original system configuration is maintained through reinitialization, progress in the system is partially or completely lost. The system is interrupted during reinitialization in response to a detected or predicted fault, error, or failure, but is not interrupted during fault-, error-, and failure-free operation. After reinitialization, the system’s ability to prevent a fault, error, or failure from affecting the correct operation of the system remains the same.

The pattern may be used in conjunction with other patterns that provide detection, containment, and mitigation in a complementary fashion, where some error/failure types are covered by the other pattern(s) and this pattern covers for the remaining error/failure types.

Performance

In the case when the monitoring system is not a part of the monitored system, the monitoring system doesn’t impact the task’s total execution time \(T_{E}\). The error/failure-free performance \(T_{f=0}\) is \(T_{E}\).

When the monitoring system is a part of the monitored system, the failure-free performance \(T_{f=0}\) of the pattern is defined by the task’s total execution time without any resilience strategy \(T_{E}\) and the time to detect or predict a fault, error or failure \(T_{d}\) with the total number of input-execute-output cycles \(P\).

The performance under errors/failures \(T_{f!=0}\) is defined by \(T_{f=0}\), plus the time to isolate the affected subsystem(s) \(T_{i}\), the time to reset the entire system or affected subsystem(s) \(T_{r}\), and the time for work lost \(T_{n}\), for each of the encountered faults, errors or failures \(N\). Assuming constant times for \(T_{i}\) and \(T_{r}\), \(T_{f!=0}\) can be defined as:

\[\begin{aligned} T_{f!=0} = T_{f=0} + N (T_{i} + T_{r}) + \sum_{n=0}^{N} T_{n} \end{aligned}\]
Reliability

Given that the pattern enables the resumption of correct operation after an error or failure, the reliability of a system employing it is defined by errors and failures that are not handled by the pattern, such as failures of the persistent storage. The reliability after applying the pattern \(R(t)\) can be obtained using the performance under failure \(T\) and the failure rate \(\lambda_{u}\) (or its inverse, the mean-time to interrupt (MTTI), \(M_{u}\)) of the unprotected part of the system.

\[\begin{aligned} R(t) = e^{-\lambda_{u} T} = e^{-T/M_{u}} \end{aligned}\]
Availability

The availability of the pattern can be calculated using the task’s total execution time without the pattern \(T_{E}\) and performance under errors/failures \(T_{f!=0}\). \(T_{E}\) is the planned uptime (PU) \(t_{pu}\). \(T_{f!=0}\) is the planned uptime (PU) \(t_{pu}\), the scheduled downtime (SD) \(t_{sd}\), and the unscheduled downtime (UD) \(t_{ud}\).

\[\begin{aligned} A = \frac{T_{E}}{T_{f!=0}} = \frac{t_{pu}}{t_{pu}+t_{ud}+t_{sd}} \end{aligned}\]
Examples

A malfunctioning compute node may be rebooted.

Rationale

The pattern prevents an existing or future fault, error, or failure from affecting the correct operation of the system through restarting the operation of the system or the affected subsystem. It relies on the ability to assume a configuration in response to a detected or predicted fault, error, or failure that retains functional equivalency with the original system configuration. The pattern may perform proactive actions, such as restarting the system or subsystem before a fault, error, or failure impacts its operation, or reactive actions, such as restarting the system or subsystem upon such an event. Progress in the system is partially or completely lost. Fault, error, or failure detection/prediction is part of the pattern. The containment and mitigation offered by this pattern are independent from the type of fault, error, or failure. The pattern has very little design complexity and has low dependence on a system’s architecture.

System Scope

In the context of INTERSECT Systems, Subsystems, and Services, this pattern can be applied to INTERSECT systems and subsystems. It would be primarily applied to an entire infrastructure system and its subsystems, but it could also be applied an entire logical system that spans across multiple infrastructure systems. It could be applied to a logical subsystem of an infrastructure system only.

Service Scope

In the context of INTERSECT Systems, Subsystems, and Services, this pattern can be applied to an INTERSECT service. If it is applied to a group of services, then this is typically within the System Scope. However, it could also be applied to interconnected services, such as to services participating in the same campaign.

Microservice Scope

In the context of the INTERSECT Microservices Architecture, this pattern can be applied to an INTERSECT microservice. If it is applied to a group of microservices, then this is typically within the Service Scope.