Technology Readiness Levels

Technology Readiness Level (TRL) is a measure used by some United States government agencies and many of the world's major companies to assess the maturity of evolving technologies (materials, components, devices, etc.) prior to incorporating that technology into a system or subsystem. Generally speaking, when a new technology is first invented or conceptualized, it is not suitable for immediate application. Instead, new technologies are usually subjected to experimentation, refinement, and increasingly realistic testing. Once the technology is sufficiently proven, it can be incorporated into a system/subsystem. The GridLAB-D project team began using TRLs in January 2011 to assess the readiness of various modules and classes for analysis work using GridLAB-D.

The primary purpose of using TRLs is to help developers and users make decisions regarding the development and deployment of GridLAB-D. The TRLs provide a common understanding of technology status and help us manage risk, make funding decisions, and assess the readiness of modules and classes for analysis projects and studies.

Depending on circumstances, there can be some disadvantages to using TRLs. In particular TRLs can result in more reporting, paperwork, reviews. It can take time for participants in GridLAB-D work to adjust to using TRLs and for the TRL process to have an effect project-wide. Finally, systems engineering processes are not addressed in the lower TRLs, which can result in difficulties transitioning technologies to higher TRLs.

For more information on the DoD definitions of TRLs and references to official publications about TRLs see Wikipedia.

= GridLAB-D TRL Definitions =

The GridLAB-D TRLs are adapted from the DOD Technology Readiness Assessment (TRA) Deskbook (July 2009)[9]. The definitions provide broad guidance on the assessment of GridLAB-D modules and classes for the purpose of modeling, simulation, and analysis.

For a system with n technologies the TRL vector defines in Equation 1 represents the state of the system with respect to the readiness of its technologies:


 * $$\mathbf{TRL_{nx1}} =

\begin{bmatrix} TRL_1 \\ &\\ TRL_2\\ &\\ ...\\ TRL_{n} \end{bmatrix} (1) $$,

where $$TRL_{i}$$ is the TRL of technology i [3].

= GridLAB-D TRL Assessment Guide =

To assist in the effective and consistent use of TRLs, a common method of technology readiness assessment is required. This section addresses the specific measures used to determine the TRL of a GridLAB-D classes.

Principle (TRL 1)
The basic requirements of the are described and the class stub created.

Concept (TRL 2)
The basic design elements of the class are described and the variables enumerated in the class.

Proof (TRL 3)
The methodology to be used has been proven on paper or numerically in other environments.

Standalone (TRL 4)
The class has been implemented and passed standalone methodological and functional validation tests.

Integrated (TRL 5)
The class has been tested in conjunction with other classes and passed integrated methodological and functional validation tests.

Demonstrated (TRL 6)
The class has been shown to produce correct results in simple exemplary situations.

Prototype (TRL 7)
The class has been shown to produce correct results in complex exemplary situations.

Qualified (TRL 8)
The class has been shown to produce correct results in realistic situations.

Proven (TRL 9)
The class has been used successfully in production-grade analysis work.

= GridLAB-D TRL Designation Guide =

Once the TRL of a class has been determined, the TRL is set in the class implementation. The GridLAB-D core uses the TRL value of each class to determine the TRL of a GLM file by calculating the lowest TRL found in the classes used by the model. The global variable technology_readiness_level is updated as model components are loaded to reflect the lowest TRL found in the GLM file.

As of Version 3.0, the method for designating the TRL of a class is exemplified in the following code


 * Note : Runtime classes are assigned level TRL_UNKNOWN=0, which is used to indicate that the TRL cannot be determined automatically.

GridLAB-D defines TRLs as follows:


 * Caveat : The method of determining class TRL is difficult to extend to modules because classes can often interact in ways that are not measured by the class TRL. Therefore the minimum TRL determine by GridLAB-D should be thought of an upper bound and the actual TRL of a model may be lower than the TRL computed by GridLAB-D.

= Integration Readiness Level =

Technology Readiness Levels (TRL) have been used by different US agencies to asses the maturity of evolving technologies before their incorporation in a system or sub-system. TRL assesses the risk associated with developing technologies, but neglects the integration links among technologies. To address this matter, Gove [1] and Sauser et all [4], [5] introduced the notion of Integration Readiness Level (IRL). IRL assesses the risk associated with the technologies integrations using a 9-levels scale [1], [5] as shown in Table 2.



Employing IRL scale in software projects shall highlight  the low levels of integration maturity and identify the area of development that requires engineering attention. The IRL index does not focus only on the physical properties of integration (i.e. standards, interfaces a.s.o.), but considers also the interaction, compatibility, reliability, quality, and performance of the integrated pieces.

Mathematically, the IRL matrix illustrates how the different technologies are integrated with each other from a system perspective. For a system with n technologies, IRL is defined in Equation 2, where $$IRL_{ij}$$ is the IRL between technologies i and j. The hypothetical integration of a technology i to itself is denoted by $$IRL_{ii}$$.


 * $$\mathbf{IRL_{nxn}} =

\begin{bmatrix} IRL_{11} & IRL_{12} & ... & IRL_{1n}\\ & \\ IRL_{21} & IRL_{22} &... & IRL_{2n} \\ & \\ ... &... &... &...\\ &\\ IRL_{n1}& IRL_{n2} &...& IRL_{nn} \end{bmatrix} (2) $$.

In GridLAB-D project the technologies are the analysis direction:
 * demand response
 * voltage reduction/control +Volt VAR
 * load modeling
 * wind turbine modeling
 * distribution feeders
 * power flow distribution.

= System Readiness Level =

System readiness Level (SRL) assess the current and future readiness of a system by incorporating both TRL and IRL. Sauser et all demonstrated how to measure a system maturity based on the TRLs and IRLs of the system under development [2]. The SRL matrix is defined as a normalized matrix of pairwise comparisons of the TRLs and IRLs [7]:
 * $$ [SRL]_{n \times 1} = [Norm]_{n \times n} \times [IRL]_{n \times n} \times [TRL]_{n \times 1} $$

It is important to note that in the cases of competing technologies, the use of normalized values provide us a more accurate way to assess system's maturity. Therefore, it is recommended to normalize the values used in the TRL and IRL matrices: the original (1,9) levels are brought to (0,1), by dividing each element by 9. The matrix $$[Norm]$$ is used to normalize the $$SRL_i$$ from $$ (0, m_i)$$ scale to (0,1) scale.

Equation 3 defines the component SRL matrix:

\begin{bmatrix} SRL_1 \\ &\\ SRL_2\\ &\\ ...\\ SRL_{n} \end{bmatrix}= \begin{bmatrix} \frac{1}{m_1} & 0 & ... & 0\\ & \\ 0 & \frac{1}{m_2} &... & 0 \\ & \\ ... &... &... &...\\ &\\ 0 & 0 &...& \frac{1}{m_n} \end{bmatrix} \times \begin{bmatrix} IRL_{11} & IRL_{12} & ... & IRL_{1n}\\ & \\ IRL_{21} & IRL_{22} &... & IRL_{2n} \\ & \\ ... &... &... &...\\ &\\ IRL_{n1}& IRL_{n2} &...& IRL_{nn} \end{bmatrix} \times \begin{bmatrix} TRL_1 \\ &\\ TRL_2\\ &\\ ...\\ TRL_{n} \end{bmatrix} = \begin{bmatrix} (IRL_{11} TRL_1 + IRL_{12} TRL_2 + ... + IRL_{1n} TRL_{n})/m_1\\ & \\ (IRL_{21} TRL_1 + IRL_{22} TRL_2 +... + IRL_{2n}TRL_n)/m_2 \\ & \\ ... \\ &\\ (IRL_{n1} TRL_1+ IRL_{n2} TRL_2 +...+ IRL_{nn} TRL_n)/m_n \end{bmatrix} (3)$$,

where
 * $$m_i$$ is the number of integrations of technology i with itself and all other technologies,
 * $$ IRL_{ij} = IRL_{ji}$$.

Equation 4 defines the composite SRL as the average of all the component SRLs:


 * $$ Composite SRL = \frac{SRL_1+ SRL_2+ ...+ SRL_n}{n}=\frac{\sum_{i=1}^n SRL_i}{n}(4)$$.

As an example, how to interpret the SRL metric for determining system maturity and status within a development lifecycle is presented for an acquisition system [8]. The ranges presented in Table 3 have been derived form sensitivity analysis with sample systems:

Inspecting the results, it is visible that a system that did not reach full maturity is able to transition into a production phase. Usually most systems are deployed without all of the technologies and integrations having reached full maturity.

= Example of SRL Calculation =

The following example is extracted form Brian Sauser et al paper "A Systems Approach To Expanding The Technology Readiness Level Within Defense Acquisition" [3]:

Consider a system with 20 technologies:


 * [[Image:System Architecture.png]]

Compute TRL and IRL for the given system:


 * [[Image: TRL.png]]
 * [[Image: IRL.png]]

Use normalization to get the component SRL and the composite SRL:
 * [[Image: SRL1.png]]


 * [[Image: SRL2.png]]
 * [[Image: SRL3.png]]

Results interpretation is illustrated below:
 * [[Image: SRL diagram.png]]

= Usability Analysis =

GridLAB-D enabled multiple types of analysis. Some of the directions are depicted in the diagram below:



= References =

[1] Gove, R., "Development Of An Integration Ontology For Systems Operational Effectiveness," School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, 2007.

[2] Sauser, B., Ramirez-Marquez, J., Henry, D., & DiMarzio, D., "A System Maturity Index For The Systems Engineering Life Cycle", International Journal of Industrial and Systems Engineering, 3(6), 2008, 673-691.

[3] Sauser, B., J. Ramirez-Marquez, R. Magnaye and W. Tan, "A Systems Approach To Expanding The Technology Readiness Level Within Defense Acquisition", International Journal of Defense Acquisition Management 1 (2008b), pp. 39-58.

[4] Sauser, B., E. Forbes, M. Long and S. McGrory, "Defining An Integration Readiness Level For Defense Acquisition," International Conference of the International Council on Systems Engineering, INCOSE, Singapore, 2009a.

[5] Sauser, B., R. Gove, E. Forbes and J. Ramirez-Marquez, "Integration Maturity Metrics: Development Of An Integration Readiness Level", Information Knowledge Systems Management 9(1) (2009b), pp 17-45.

[6] Sauser, B. and J. Ramirez-Marquez, "System Development Planning Via System Maturity Optimization", IEEE Transaction on Engineering Management 56 (2009c), no. 3, pp. 533-548.

[7] Weiping Tan, Jose Ramirez-Marquez, and Brian Sauser, "A Probabilistic Approach to System Maturity Assessment", Wiley Online Library, DOI 10.1002/sys.20179

[8] DoD, "Defense Acquisition Guidebook", Directive 5000.2, Department of Defense, Washington, DC, 2005a.

[9] DoD, "Technology Readiness Assessment (TRA) deskbook", DUSD (S&T), U.S. Department of Defense, Washington, DC, 2009.