validate - Command line option to run model validation test suite Template:NEW30
Synopsis
host% gridlabd test-options --validate run-options
Description
The validate command line option initiates the standard model validation process for GridLAB-D. The process functions as follows:
- Recursively scan the working directory for directories called autotest, ignoring any directory that contains a file named validate.no.
- For each GLM files found in an autotest directory:
- Create a folder by the same name (or purge existing folder if the clean global variable is defined)
- Copy the GLM file into the folder
- Run gridlabd on the GLM file with all the run-options
- Time and observe results (success, fail, exception)
- Report total time and results
The test-options are applied to the validation instance of GridLAB-D, while the run-options are applied to the test instance of GridLAB-D. For example,
host% gridlabd --threadcount 2 --validate
will run two validation tests simultaneously, while
host% gridlabd --validate --threadcount 2
will run a single two-threaded validation test at a time.
Output
Validation test failures are reported as errors. Optional test failures are reported as warning.
By default all output is redirected to the default output streams (see redirect). You can send all output to the console using --redirect none command option.
Options
The validate procedure is controlled using the following global variables.
- --define|-D clean=1
- If the clean variable is defined, the directories used to run each test are purged before the test is started.
- --define|-D validate=NONE|TRUN|TERR|TEXC|TOPT|TSTD|TALL|RDIR|RGLD|RALL
- Controls which validation tests are performed.
- NONE
- None of the tests are performed (provided to definition purposes only)
- TRUN
- Includes tests that are expected to succeed.
- TERR
- Includes tests that are expected to cause errors.
- TEXC
- Includes tests that are expected to cause exceptions.
- TOPT
- Includes tests that are optional
- TSTD
- Includes all standard tests (TRUN|TERR|TEXC)
- TALL
- Includes all tests (TRUN|TERRT|EXC|TOPT)
- RDIR
- Includes directory scan results in output report
- RGLM
- Includes GLM file results in output report
- RALL
- Includes both directory and file results in output report
- --define|-D force_validate=1
- Overrides the effect of the validate.no file if found in a folder. Normally, validate does not process folders that contain the file validate.no.
Report Format
When a validation run is performed only progress and unexpected results are normally displayed on screen. However, a detailed validation report is output to the file validate.csv. You may change the name and type of the validation report file by setting the validate_report global variable. For example,
host% gridlabd -D validate_report=validate.txt --validate
will output a (tab-delimited) TXT file instead of a CSV file.
The validation report is formatted to facilitate parsing by scripts and includes the following sections.
- TEST CONFIGURATION
- This section provide general information about the gridlabd environment configuration used to perform the validation test.
- DIRECTORY SCAN RESULTS
- This section lists how many files were found in each directory scanned. This section is only output if the RDIR or RALL validate (global) options are set.
- FILE TEST RESULTS
- This section lists the results of each test and is only output if the RGLM or RALL validate (global) options are set. In the first column S indicates the test unexpectedly succeeded, E indicates the test unexpectedly produced an error, and X indicates the test unexpectedly produces an exception. The second column provides the runtime of the test. The third column provides the name of the GLM file tested.
- OVERALL RESULTS
- The section summarizes the overall results of the validation test. Any unusual results are flagged with a triple exclamation point (!!!) in the first column. The result code is a unique number that is generated that can be compared to validation test results from other platforms. A result code having all zeros represents complete success. Non-zero values correspond to various failure modes on specific files.
Mailing Reports
- Linux/Mac only
- When a validation run is completed, a copy of the validation report can be emailed using the mailto global variable. For example
host% gridlabd -D mailto user@localhost --validate
- Will email the validation report in to user on the local machine.
- Note
- Your local mail server must be configured properly to deliver email to remote hosts and your mail client must be configured properly to read mail on the local mail server. GridLAB-D cannot detect either mail server or mail client configuration errors. Thus problems with either of these can cause mailed report delivery errors to occur with an error being reported in GridLAB-D.
Caveats
Multithreaded operation can cause intermingled output, particularly when used in conjunction with --redirect none. This is due to the lack of locking in the output message streams when running multiple jobs on a single console.
Some output from gridlabd runs cannot be redirected and will always be displayed on the console. Known problems include output from compilers, linked applications, and scripts.
Bugs
Unhandled exceptions in Windows can cause modal dialogs to pop up that block the process. This behavior appears to be impossible to suppress. See http://sourceforge.net/p/gridlab-d/tickets/606 for details.
Version
The built-in validate option was introduced in Hassayampa (Version 3.0).