Technology Readiness Level (TRL) is a measure used by some United States government agencies and many of the world's major companies to assess the maturity of evolving technologies (materials, components, devices, etc.) prior to incorporating that technology into a system or subsystem. Generally speaking, when a new technology is first invented or conceptualized, it is not suitable for immediate application. Instead, new technologies are usually subjected to experimentation, refinement, and increasingly realistic testing. Once the technology is sufficiently proven, it can be incorporated into a system/subsystem. The GridLAB-D project team began using TRLs in January 2011 to assess the readiness of various modules and classes for analysis work using GridLAB-D.
The primary purpose of using TRLs is to help developers and users make decisions regarding the development and deployment of GridLAB-D. The TRLs provide a common understanding of technology status and help us manage risk, make funding decisions, and assess the readiness of modules and classes for analysis projects and studies.
Depending on circumstances, there can be some disadvantages to using TRLs. In particular TRLs can result in more reporting, paperwork, reviews. It can take time for participants in GridLAB-D work to adjust to using TRLs and for the TRL process to have an effect project-wide. Finally, systems engineering processes are not addressed in the lower TRLs, which can result in difficulties transitioning technologies to higher TRLs.
For more information on the DoD definitions of TRLs and references to official publications about TRLs see Wikipedia.
GridLAB-D TRL Definitions
The GridLAB-D TRLs are adapted from the DOD Technology Readiness Assessment (TRA) Deskbook (July 2009)[9]. The definitions provide broad guidance on the assessment of GridLAB-D modules and classes for the purpose of modeling, simulation, and analysis.
| TRL Id | Definition of Technology Readiness Level | Description |
|---|---|---|
| 1. | Basic principles described | Lowest level of technology readiness. Basic research begins to be translated into applied research and development. Example might include paper studies of a technology's basic properties.
SW: Lowest level of software readiness. Basic research begins to be translated into applied research and development. Examples might include a concept that can be implemented in software or analytic studies of an algorithm’s basic properties. |
| 2. | Application concept formulated | Invention has begun. Basic principles are described and practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support assumptions or expectations. Examples are still limited to paper studies. |
| 3. | Analytical proof of concept | Active research and development is initiated. This includes computational studies to numerically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative.
SW: Active research and development is initiated. This includes analytical studies to produce code that validates analytical predictions of separate software elements of the technology. Examples include soft-ware components that are not yet integrated or representative but satisfy an operational need. Algorithms run on a surrogate processor in a laboratory environment. |
| 4. | Standalone component validated | Basic technological components are integrated in prototype form to establish that the pieces will work together. This is "low fidelity" compared to the eventual system. Examples include integration of ad hoc modules or classes in trivial system models.
SW: Basic software components are integrated to establish that they will work together. They are relatively primitive with regard to efficiency and reliability compared to the eventual system. System software architecture development initiated to include interoperability, reliability, maintainability, extensibility, scalability, and security issues. Software integrated with simulated current/legacy elements as appropriate. |
| 5. | Integrated component validated | Fidelity of prototype technology shown to function properly in relevant environments. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a controlled environment. Examples include integration of components in simplified system models.
SW: Reliability of software ensemble increases significantly. The basic software components are integrated with reasonably realistic supporting elements so that it can be tested in a simulated environment. Examples include “high fidelity” laboratory integration of software components. System software architecture established. Algorithms run on a processor(s) with characteristics expected in the operational environment. Software releases are “Alpha” versions and configuration control is initiated. Verification, Validation, and Accreditation (VV&A) initiated.
|
| 6. | System/subsystem demonstrated | Representative model or prototype system, which is well beyond the prototype tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology's demonstrated readiness. Examples include testing a prototype module in complex system models.
SW: Representative model or prototype system, which is well beyond that of TRL 5, is tested in a relevant environment. Represents a major step up in software-demonstrated readiness. Examples include testing a prototype in a live/virtual experiment or in a simulated operational environment. Algorithms run on processor of the operational environment are integrated with actual external entities. Software releases are “Beta” versions and configuration controlled. Software sup-port structure is in development. VV&A is in process. |
| 7. | Prototype demonstrated | Prototype functions in an operational environment similar to or in actual planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in full system model. Examples include testing the prototype on the IEEE 8500 bus model.
SW: Represents a major step up from TRL 6, re-quiring the demonstration of an actual system prototype in an operational environment, such as in a command post or air/ground vehicle. Algorithms run on processor of the operational environment are integrated with actual external entities. Software support structure is in place. Software releases are in distinct versions. Frequency and severity of software deficiency reports do not significantly degrade functionality or performance. VV&A completed. |
| 8. | System 'analysis qualified' | Technology has been proven through thorough test and demonstration to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended analysis environment to determine if it meets design specifications.
SW: Software has been demonstrated to work in its final form and under expected conditions. In most cases, this TRL represents the end of system development. Examples include test and evaluation of the software in its intended system to determine if it meets design specifications. Software releases are production versions and configuration controlled, in a secure environment. Software deficiencies are rapidly resolved through support infrastructure. |
| 9. | System proven | Actual application of the technology in its final form and under project delivery conditions, such as those encountered in analysis project. In almost all cases, this is the end of the last "bug fixing" aspects of true system development. Examples include using the system for a client's analysis task.
SW: Actual application of the software in its final form and under mission conditions, such as those encountered in operational test and evaluation. In almost all cases, this is the end of the last “bug fixing” aspects of the system development. Examples include using the system under operational mission conditions. Software releases are production versions and configuration controlled. Frequency and severity of software deficiencies are at a minimum. |
| TRL Id | Definition of Technology Readiness Level | Description |
|---|---|---|
| 0. | Not applicable | Cannot be determined. |
| 1. | Basic principles described (mathematical formulation) | Lowest level of software readiness. Basic research begins to be translated into applied research and development by providing a detailed mathematical formulation. Examples might include a concept that can be implemented in software or analytic studies of an algorithm’s basic properties. |
| 2. | Application concept formulated (algorithm) | Invention has begun. Basic individual algorithms or functions are prototyped and documented. Results are speculative and there is no proof or detailed analysis to support assumptions or expectations. Examples are still limited to paper studies. |
| 3. | Analytical proof of concept (prototype) | Active research and development is initiated. Depending on the size and complexity of the implementation:
• a basic prototype of the integrated critical system has been designed, build and tested; the whole development process has been documented, • analytic studies to produce code that validates analytical predictions of separate software elements of the technology are done. Examples include implementation of software components that are not yet integrated or representative but satisfy an operational need. Algorithms are run and tested on a surrogate processor in a laboratory environment. |
| 4. | Standalone component validated (earliest version) | Basic software components are integrated to establish that they will work together. They are relatively primitive with regard to efficiency and reliability compared to the eventual system. System software architecture development initiated to include interoperability, reliability, maintainability, extensibility, scalability, and security issues. Software integrated with simulated current/legacy elements as appropriate. Verification and Validation process is partially completed, or completed for only a subset of the functionality in a representative simulated laboratory environment. Documentation is as for TRL3 plus: User Manual and Design Documents. |
| 5. | Integrated component validated (ALPHA version) | Reliability of software ensemble increases significantly. All software components are integrated with reasonably realistic supporting elements so that the software can be tested and completely validated in a simulated environment. Examples include “high fidelity” laboratory integration of software components. System software architecture established. Algorithms run on a processor(s) with characteristics expected in the operational environment. Software releases are “Alpha” versions and configuration control is initiated. Full documentation according to the applicable software standards, test plans and application examples, including all use cases and error handling should be provided. |
| 6. | System/subsystem demonstrated (BETA version) | Representative model or prototype system, which is well beyond that of TRL 5, is tested in a relevant environment. Represents a major step up in software-demonstrated readiness. Examples include testing a prototype in a live/virtual experiment or in a simulated operational environment. Algorithms run on processor of the operational environment are integrated with actual external entities. Software releases are “Beta” versions and configuration controlled. Configuration control and Quality assurance processes are fully deployed. Verification and Validation process is completed for the intended scope (including robustness) and the system is validated in an end to-end fully representative laboratory environment (including real target). |
| 7. | Prototype demonstrated (product RELEASE) | Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in an operational environment, such as in a command post or air/ground vehicle. Algorithms run on processor of the operational environment are integrated with actual external entities. Software support structure is in place. Software releases are in distinct versions. Frequency and severity of software deficiency reports do not significantly degrade functionality or performance. Verification and Validation is completed, validity of solution is confirmed within intended application. Requirements specification validated by the users. Engineering support and maintenance organization, including helpdesk, are in place. |
| 8. | System 'analysis qualified' (general product) | Software has been demonstrated to work in its final form and under expected conditions. In most cases, this TRL represents the end of system development. Examples include test and evaluation of the software in its intended system to determine if it meets design specifications. Software releases are production versions and configuration controlled, in a secure environment. Software deficiencies are rapidly resolved through support infrastructure. Provide full documentation including specifications, design definition, design justification, verification and validation (qualification file), users and installation manuals and software problem reports and non-compliances. |
| 9. | System proven (live product) | Actual application of the software in its final form and under mission conditions, such as those encountered in operational test and evaluation. In almost all cases, this is the end of the last “bug fixing” aspects of the system development. Examples include using the system under operational mission conditions. Software releases are production versions and configuration controlled. Frequency and severity of software deficiencies are at a minimum. Sustaining engineering, including maintenance and upgrades are in place Updates to documentation and qualification files are in place. |
For a system with n technologies the TRL vector defines in Equation 1 represents the state of the system with respect to the readiness of its technologies:
- <math>\mathbf{TRL_{nx1}} =
\begin{bmatrix} TRL_1 \\ &\\ TRL_2\\ &\\ ...\\ TRL_{n} \end{bmatrix} (1) </math>,
where <math>TRL_{i}</math> is the TRL of technology i [3].
GridLAB-D TRL Assessment Guide
To assist in the effective and consistent use of TRLs, a common method of technology readiness assessment is required. This section addresses the specific measures used to determine the TRL of a GridLAB-D classes.
Principle (TRL 1)
The basic requirements of the are described and the class stub created.
Concept (TRL 2)
The basic design elements of the class are described and the variables enumerated in the class.
Proof (TRL 3)
The methodology to be used has been proven on paper or numerically in other environments.
Standalone (TRL 4)
The class has been implemented and passed standalone methodological and functional validation tests.
Integrated (TRL 5)
The class has been tested in conjunction with other classes and passed integrated methodological and functional validation tests.
Demonstrated (TRL 6)
The class has been shown to produce correct results in simple exemplary situations.
Prototype (TRL 7)
The class has been shown to produce correct results in complex exemplary situations.
Qualified (TRL 8)
The class has been shown to produce correct results in realistic situations.
Proven (TRL 9)
The class has been used successfully in production-grade analysis work.
GridLAB-D TRL Designation Guide
Once the TRL of a class has been determined, the TRL is set in the class implementation. The GridLAB-D core uses the TRL value of each class to determine the TRL of a GLM file by calculating the lowest TRL found in the classes used by the model. The global variable technology_readiness_level is updated as model components are loaded to reflect the lowest TRL found in the GLM file.
As of Version 3.0, the method for designating the TRL of a class is exemplified in the following code
oclass = gl_register_class(module,"example",sizeof(example),passconfig);
if (oclass==NULL)
throw "unable to register class 'example'";
else
oclass->trl = TRL_STANDALONE; // TRL 4
- Note
- Runtime classes are assigned level TRL_UNKNOWN=0, which is used to indicate that the TRL cannot be determined automatically.
GridLAB-D defines TRLs as follows:
typedef enum {
TRL_UNKNOWN = 0,
TRL_PRINCIPLE = 1,
TRL_CONCEPT = 2,
TRL_PROOF = 3,
TRL_STANDALONE = 4,
TRL_INTEGRATED = 5,
TRL_DEMONSTRATED = 6,
TRL_PROTOTYPE = 7,
TRL_QUALIFIED = 8,
TRL_PROVEN = 9,
} TECHNOLOGYREADINESSLEVEL;
- Caveat
- The method of determining class TRL is difficult to extend to modules because classes can often interact in ways that are not measured by the class TRL. Therefore the minimum TRL determine by GridLAB-D should be thought of an upper bound and the actual TRL of a model may be lower than the TRL computed by GridLAB-D.
Integration Readiness Level
Technology Readiness Levels (TRL) have been used by different US agencies to asses the maturity of evolving technologies before their incorporation in a system or sub-system. TRL assesses the risk associated with developing technologies, but neglects the integration links among technologies. To address this matter, Gove [1] and Sauser et all [4] , [5] introduced the notion of Integration Readiness Level (IRL). IRL assesses the risk associated with the technologies integrations using a 9-levels scale [1], [5] as shown in Table 2.
| IRL | Definition | Description |
|---|---|---|
| 9 | Integration is Mission Proven through successful mission operations. | IRL 9 represents the integrated technologies being used in the system environment successfully. In order for a technology to move to the TRL 9, it must first be integrated into the system and then proven in the relevant environment; thus, progressing IRL to 9 also implies maturing the component technology to the TRL 9. |
| 8 | Actual integration completed and Mission Qualified through test and demonstration in the system environment. | IRL 8 represents not only the integration-meeting requirements, but also a system-level demonstration in the relevant environment. This will reveal any unknown bugs/defects that could not be discovered until the interaction of the two integrating technologies was observed in the system environment. |
| 7 | The integration of technologies has been Verified and Validated with sufficient detail to be actionable. | IRL 7 represents a significant step beyond IRL 6; the integration has to work from a technical perspective, but also from a requirements perspective. IRL 7 represents the integration meeting requirements such as performance, throughput, and reliability. |
| 6 | The integrating technologies can Accept, Translate, and Structure Information for its intended application. | IRL 6 is the highest technical level to be achieved; it includes the ability to not only control integration, but to specify what information to exchange, to label units of measure to specify what the information is, and the ability to translate from a foreign data structure to a local one. |
| 5 | There is sufficient Control between technologies necessary to establish, manage, and terminate the integration. | IRL 5 simply denotes the ability of one or more of the integrating technologies to control the integration itself; this includes establishing, maintaining, and terminating. |
| 4 | There is sufficient detail in the Quality and Assurance of the integration between technologies. | Many technology-integration failures never progress past IRL 3, due to the assumption that if two technologies can exchange information successfully, then they are fully integrated. IRL 4 goes beyond simple data exchange and requires that the data sent is the data received and there exists a mechanism for checking it. |
| 3 | There is Compatibility (i.e., common language) between technologies to orderly and efficiently integrate and interact. | IRL 3 represents the minimum required level to provide successful integration. This means that the two technologies are able to not only influence each other, but also to communicate interpretable data. IRL 3 represents the first tangible step in the maturity process. |
| 2 | There is some level of specificity to characterize the Interaction (i.e., ability to influence) between technologies through their interface. | Once a medium has been defined, a “signaling” method must be selected such that two integrating technologies are able to influence each other over that medium. Since IRL 2 represents the ability of two technologies to influence each other over a given medium, this represents integration proof-of-concept. |
| 1 | An Interface between technologies has been identified with sufficient detail to allow characterization of the relationship. | This is the lowest level of integration readiness and describes the selection of a medium for integration. |
Employing IRL scale in software projects shall highlight the low levels of integration maturity and identify the area of development that requires engineering attention. The IRL index does not focus only on the physical properties of integration (i.e. standards, interfaces a.s.o.), but considers also the interaction, compatibility, reliability, quality, and performance of the integrated pieces.
Mathematically, the IRL matrix illustrates how the different technologies are integrated with each other from a system perspective. For a system with n technologies, IRL is defined in Equation 2, where <math>IRL_{ij}</math> is the IRL between technologies i and j. The hypothetical integration of a technology i to itself is denoted by <math>IRL_{ii}</math>.
- <math>\mathbf{IRL_{nxn}} =
\begin{bmatrix}
IRL_{11} & IRL_{12} & ... & IRL_{1n}\\
& \\
IRL_{21} & IRL_{22} &... & IRL_{2n} \\
& \\
... &... &... &...\\
&\\
IRL_{n1}& IRL_{n2} &...& IRL_{nn}
\end{bmatrix} (2) </math>.
In GridLAB-D project the technologies are the analysis direction:
- demand response
- voltage reduction/control +Volt VAR
- load modeling
- wind turbine modeling
- distribution feeders
- power flow distribution.
System Readiness Level
System readiness Level (SRL) assess the current and future readiness of a system by incorporating both TRL and IRL. Sauser et all demonstrated how to measure a system maturity based on the TRLs and IRLs of the system under development [2]. The SRL matrix is defined as a normalized matrix of pairwise comparisons of the TRLs and IRLs [7]:
- <math> [SRL]_{n \times 1} = [Norm]_{n \times n} \times [IRL]_{n \times n} \times [TRL]_{n \times 1} </math>
It is important to note that in the cases of competing technologies, the use of normalized values provide us a more accurate way to assess system's maturity. Therefore, it is recommended to normalize the values used in the TRL and IRL matrices: the original (1,9) levels are brought to (0,1), by dividing each element by 9.
The matrix <math>[Norm]</math> is used to normalize the <math>SRL_i</math> from <math> (0, m_i)</math> scale to (0,1) scale.
Equation 3 defines the component SRL matrix:
- <math>
\begin{bmatrix} SRL_1 \\ &\\ SRL_2\\ &\\ ...\\ SRL_{n} \end{bmatrix}= \begin{bmatrix}
\frac{1}{m_1} & 0 & ... & 0\\
& \\
0 & \frac{1}{m_2} &... & 0 \\
& \\
... &... &... &...\\
&\\
0 & 0 &...& \frac{1}{m_n}
\end{bmatrix} \times \begin{bmatrix}
IRL_{11} & IRL_{12} & ... & IRL_{1n}\\
& \\
IRL_{21} & IRL_{22} &... & IRL_{2n} \\
& \\
... &... &... &...\\
&\\
IRL_{n1}& IRL_{n2} &...& IRL_{nn}
\end{bmatrix} \times \begin{bmatrix} TRL_1 \\ &\\ TRL_2\\ &\\ ...\\ TRL_{n} \end{bmatrix} = \begin{bmatrix}
(IRL_{11} TRL_1 + IRL_{12} TRL_2 + ... + IRL_{1n} TRL_{n})/m_1\\
& \\
(IRL_{21} TRL_1 + IRL_{22} TRL_2 +... + IRL_{2n}TRL_n)/m_2 \\
& \\
... \\
&\\
(IRL_{n1} TRL_1+ IRL_{n2} TRL_2 +...+ IRL_{nn} TRL_n)/m_n
\end{bmatrix} (3)</math>,
where
- <math>m_i</math> is the number of integrations of technology i with itself and all other technologies,
- <math> IRL_{ij} = IRL_{ji}</math>.
Equation 4 defines the composite SRL as the average of all the component SRLs:
- <math> Composite SRL = \frac{SRL_1+ SRL_2+ ...+ SRL_n}{n}=\frac{\sum_{i=1}^n SRL_i}{n}(4)</math>.
As an example, how to interpret the SRL metric for determining system maturity and status within a development lifecycle is presented for an acquisition system [8]. The ranges presented in Table 3 have been derived form sensitivity analysis with sample systems:
| SRL | Definition | Description |
|---|---|---|
| 0.90 to 1.00 | Operations & Support | Execute a support program that meets operational support performance requirements and sustains the system in the most cost-effective manner over its total lifecycle. |
| 0.80 to 0.89 | Production | Achieve operational capability that satisfies mission needs. |
| 0.60 to 0.79 | System Development & Demonstration | Develop system capability or (increments thereof); reduce integration and manufacturing risk; ensure operational supportability; reduce logistics footprint; implement human systems
integration; design for production; ensure affordability and protection of critical program information; and demonstrate system integration, interoperability, safety and utility. |
| 0.40 to 0.59 | Technology Development | Reduce technology risks and determine appropriate set of technologies to integrate into a full system. |
| 0.10 to 0.39 | Concept Refinement | Refine initial concept; develop system/technology strategy. |
Inspecting the results, it is visible that a system that did not reach full maturity is able to transition into a production phase. Usually most systems are deployed without all of the technologies and integrations having reached full maturity.
Example of SRL Calculation
The following example is extracted form Brian Sauser et al paper "A Systems Approach To Expanding The Technology Readiness Level Within Defense Acquisition" [3]:
Consider a system with 20 technologies:
Compute TRL and IRL for the given system:
Use normalization to get the component SRL and the composite SRL:
Results interpretation is illustrated below:
Usability Analysis
GridLAB-D enabled multiple types of analysis. Some of the directions are depicted in the diagram below:
References
[1] Gove, R., "Development Of An Integration Ontology For Systems Operational Effectiveness," School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, 2007.
[2] Sauser, B., Ramirez-Marquez, J., Henry, D., & DiMarzio, D., "A System Maturity Index For The Systems Engineering Life Cycle", International Journal of Industrial and Systems Engineering, 3(6), 2008, 673-691.
[3] Sauser, B., J. Ramirez-Marquez, R. Magnaye and W. Tan, "A Systems Approach To Expanding The Technology Readiness Level Within Defense Acquisition", International Journal of Defense Acquisition Management 1 (2008b), pp. 39-58.
[4] Sauser, B., E. Forbes, M. Long and S. McGrory, "Defining An Integration Readiness Level For Defense Acquisition," International Conference of the International Council on Systems Engineering, INCOSE, Singapore, 2009a.
[5] Sauser, B., R. Gove, E. Forbes and J. Ramirez-Marquez, "Integration Maturity Metrics: Development Of An Integration Readiness Level", Information Knowledge Systems Management 9(1) (2009b), pp 17-45.
[6] Sauser, B. and J. Ramirez-Marquez, "System Development Planning Via System Maturity Optimization", IEEE Transaction on Engineering Management 56 (2009c), no. 3, pp. 533-548.
[7] Weiping Tan, Jose Ramirez-Marquez, and Brian Sauser, "A Probabilistic Approach to System Maturity Assessment", Wiley Online Library, DOI 10.1002/sys.20179
[8] DoD, "Defense Acquisition Guidebook", Directive 5000.2, Department of Defense, Washington, DC, 2005a.
[9] DoD, "Technology Readiness Assessment (TRA) deskbook", DUSD (S&T), U.S. Department of Defense, Washington, DC, 2009.