TestCocoon
Open Source
Code Coverage Measurement for C/C++ and C#
Part II |
| Execution Name | * |
TESTS using the wildcard syntax:| Execution Name | TESTS/* |
TESTS using the regular expression syntax:| Execution Name | =TESTS/.* |
Here some examples of renaming execution:
-
To put all executions in the directory
TESTSset:
Actual Execution Name =.*New Execution Name TESTS/& - To move all executions in the directory
TESTSin the directoryOLDset:
Actual Execution Name =TESTS/(.*)New Execution Name OLD/\1 - To rename all executions in all directories in
testname [directory]set:
Actual Execution Name =([^/]*)/([^/]*)New Execution Name \2 [\1]
5.2 Loading an Execution Report
The execution report is produced upon application exit.
Its name is defined by the initialization function
__coveragescanner_install of the CoverageScanner library
(see chap. coveragescanner_install).
Its extension is followed by ".csexe".
It contains the list of all executed code segments for each application run.
The execution report is never overwritten but
execution contents are appended.
To load an execution report click on "File->Load Execution Report"
or the icon 
on the toolbar.
The dialog as shown in fig:coveragebrowser_loadexec_fig will appear.
The file can be loaded directly or using a script.
To load directly, click on "file",
enter the path of the ".csexe" file to load
to the free form input box or use the browse button.
If the option "Delete after transferring" is selected, the execution report file
will be deleted after transfer.
The option "When file becomes modified" permits to reopen this dialog
automatically when the execution report is modified.
The "Name" field allows the user to specify the name of the imported instrumentation
if this is not specified in the execution report (see chap. executionreportname). Also, it is
possible to set the execution status (passed, failed or to be checked manually) of all imported executions.
If more than one instrumentation is imported an index is appended to this default name.
The option "Import Preprocessing" permits to select the behaviour
in the case of conflicts or redundant executions:
- "Ignore Duplicate Executions"
- Executions which have executed the same code than an execution which are already imported are ignored.
- "Import Duplicate Executions"
- Executions are imported if at least one instrumented source code line is executed.
- "Import Duplicate And Empty Executions"
- All executions are imported.
- "Merge all executions with the same name together"
- All executions with the same name are merged (i.e. the execution count of each instrumentation is added).
Invalid executions are not imported and
a summary shows when the operation is completed (see figure fig:coveragebrowser_loadreport)
when "Display import summary" option is selected.
If the execution report is not accessible through the file system,
a script can be used. The script has only to print the contain of
the execution report to the standard output (stdout). The standard error
output (stderr) is displayed on the screen and can be used for debugging
purpose. On success, the script must exit with the value 0.
5.3 Test Benefit Analysis Mode
The test benefit analysis mode if activated by clicking on the button 
.
In this mode, CoverageBrowser does not display the coverage of a set of tests but the lines
covered by an execution which are not covered by a set of reference executions.
In other words, CoverageBrowser shows what an execution is covering more than a set of other executions.
The list of reference executions are the executions checked in the execution list.
Double-clicking on the name of the execution permits to select the test to analyse.
Only the instrumented lines which are executed by this execution are shown, the other are in the state "hidden".
Also the coverage statistics displayed in the source list are only containing the percentage of
instrumented statements which are only executed by this the selected execution.
Chapter 6 Source Browser Window
The "Source Browser" window can be displayed by clicking
on "View->Source Browser".
Each item is a C/C++ source file and the sub-entries are the list
of included headers which have been instrumented. When an item
the "Source Window" will be displayed.
The "Source Browser" window displays rudimentary code coverage statistics
for each source code file. The color of each item is selected according to code coverage
statistics for each file and the watermarks.(see chap. watermarks)
The input field 
permits to filter the output using regular expressions
(see chap. sec:coveragebrowser-filter).
The item text which can be filtered is the full path of the source file. (example: c:\directory\file.cpp)
Chapter 7 Method Browser Window
The "Method Browser" window can be opened by clicking
on "View->Method Browser".
The "Method Browser" window displays a the code coverage
statistics of each C/C++ functions, classes and namespaces.
Clicking on a function permits to show all instrumented lines of it
in the "Source Viewer".
The input field permits to filter the output using regular expressions
(see chap. sec:coveragebrowser-filter).
The item text which can be filtered is the symbol name including the class name and the namespace.
(example: MyNamespace::MyClass::MyProc)
Chapter 8 Source Viewer Window
8.1 Source Display
The "Source Viewer" window can be displayed by clicking on "View->New Source Window".
The "Source Viewer" window displays the source file or its C/C++ preprocessed view.
Clicking on 
enables the user to toggle between the 2 different views.
The source code is colored with code coverage instrumentations.
The colors used are described in section sec:color_convension.
By selecting an area with the mouse, corresponding instrumentations are highlighted and a
detailed description of them is displayed in the "Explanation"
window (see chap. code-coverage-explanation). It is possible to navigate between
instrumentations using the navigation buttons 
and 
.
Navigation buttons in yellow, blue, red and green permit to jump to the next or previous
comments, manually validated instrumentations, non-executed code parts or executed code parts.
Clicking on the source code selects the nearest instrumentation.
If a comment is entered for an instrumentation, the icon 
is displayed in the margin.
On the right side, CoverageBrowser displays the test coverage
count1
or the code coverage count2
for each line.
If a source code line contains more than one instrumentation,
CoverageBrowser display the range of their counts.
8.2 Color Convention
Instrumentations are displayed in a source window using different colors:
- Green - "Executed"
- An instrumentation is displayed in green when the code has been executed.
- Orange - "Partially Executed"
- An instrumentation is marked as "Partially Executed" when it is not completely executed. This occurs when a Boolean expression was only true or false for example. In the case of a source code line which contains more than one instrumentation, the line is marked as "Partially Executed" when one of its instrumentations has not been "Executed". A detailed information is displayed in the "Explanation" window (see chap. code-coverage-explanation).
- Red - "Never Executed" or "Execution count too low"
- An instrumentation is displayed in red when the code is never executed or when the execution count is lower that than the execution count requested.
- Blue - "Manually Set To Be Executed"
- The user has the possibility to mark an instrumentation as ’Manually Validated’. This is usually to exclude dead code or code which cannot be tested for code coverage statistics. This state is only relevant if executions are in a "Never Executed" or "Partially Executed" state.
- Gray - "Unknown" or "Hidden"
- Gray is used when no information about instrumentation is available. This occurs when no executions are selected or when analysing the benefit of tests (see chap. test-benefit).
- Magenta - "Registration Needed"
- Instrumentations displayed in magenta require product registration to be visible. The evaluation version only displays a subset of the instrumentations. See chapter order to register TestCocoon.
8.3 Comments
8.3.1 Editing Comments
It is possible to add a comment by selecting an instrumentation and clicking on
the context menu entry "Add/Edit Comment", the main menu entry
"instrumentation->Add/Edit Comment" or the icon on the toolbar.
The "Comment" Window fig:coveragebrowser_comment_edt appears and allows a comment to be edited. The most recently entered comments can be retrieved by clicking on the "Last Comments" selection field. Basic text formatting is possible using the integrated toolbar buttons (see comments_shortcut).
The comment is printed in the explanation in a yellow box and the icon ()
is displayed in the source window near the line number.
8.3.2 Removing Comments
It is possible to remove a comment by selecting an instrumentation
and clicking on the context menu entry "Remove Comment", the main menu
entry "instrumentation->Remove Comment" or the icon 
on the toolbar.
Chapter 9 Code Coverage Explanation Window
The "Explanation" Window fig:coveragebrowser_expl_fig
is a docking window which is automatically updated with a detailed
description of the selected instrumentations of the source window.
For each instrumentation, the following information is displayed:
- A short description of the instrumentation state (see chap. sec:color_convension).
- The preprocessed source code which is concerned by the instrumentation.
- For Boolean expressions, the truth-table which shows executed and unexecuted states.
- The list of executions which are executing the portion of code.
- User comments.
CoverageBrowser displays the truth-table in the case of a Boolean expression which is partially executed.
The truth-table indicates which value the expression has or has not reached during execution.
Example: the truth-table truth_table indicates that the expression was false but not true.
Chapter 10 Statistic Window
The "Statistic" Window fig:coveragebrowser_statistics
is a docking window which is automatically updated with the
code coverage statistic for the whole project.
If the coverage level is greater than one, the "Statistic" Window displays
the statistics of the current level and the level one.
Chapter 11 Filter using wildcard expression or regular expressions
CoverageBrowser provides a generic filtering mechanism of rows using wildcard or regular expressions.
Wildcard expressions are activated per default and regular expressions are selected when the expression starts with
an equal sign (’=’).
Clicking on the filter icon converts the expression from wildcard into regular form as far as this is possible
and vice versa.
11.1 Wildcard Expression
|
Example: foo*bar match any tests containing the string foo followed by bar.
11.2 Regular Expression
The first character must be ’=’ to activate the regular expressions.
11.2.1 Pattern matching
|
11.2.2 String substitution
|
Chapter 12 Code/Test Coverage Level
The menu entry "Instrumentation->Level:x" permits to set the targeted code coverage count or, if the compiled with instrumentation hit support1, the targeted test coverage count.
The level is corresponding of the number of code/test coverage count necessary
to consider that an instrumented code is executed.
Example: Setting the level to 10, will made necessary to execute 10 times
the each line of the source code if compiled with code coverage count.
If compiled with code coverage hit, 10 execution runs need to execute
each lines of the source code.
The menu entry "Tools->Test Coverage Count Mode" and the button 
permits to switch between code coverage count and test coverage count analysis. This simulates the behaviour of the compilation with code coverage hit support2 when the project is compiled with code coverage count support3.
Chapter 13 Code Coverage Algorithm
CoverageBrowser displays the code coverage analysis (branch, decision or condition) generated be CoverageScanner.
But "Instrumentation->Coverage Method->Branch only" permits to reduce the analysis to the code coverage of branches.
This produces the same result as compiling with the --cs-branch of CoverageScanner.
"Instrumentation->Coverage Method->Decision, Condition and Branches" permits to show the code coverage analysis at the level defined at the compilation.
Here a short overview of the command line options necessary for each code coverage analysis method:
|
Chapter 14 Optimized Execution Order
CoverageBrowser is able to calculate an optimized order of the executions
(i.e.: the order of tests which permits to maximize the code coverage after
each execution).
This order is specially adapted to manual testing: following this order permits
to execute first the tests which are giving a high code coverage and so detecting rapidly
errors in the first test executions.
To calculate the execution order proceed as follows:
- Select a set of executions in the "Executions" window.
- Click on "Tools->Optimized Execution Order…". The window fig:optimized_execution_order_fig will de display.
Chapter 15 Comparing Code Coverage of Two Software Releases
CoverageBrowser is able to instrumentation database together in order to:
- check is the modified/unmodified code is correctly tested.
- find which tests are impacted by a source code modification.
This feature is particularly adapted to compare two releases together which contains small modifications (bug fixes only) and to limit the tests of the modified code only.
In this mode CoverageBrowser uses the following typographic rules:
CoverageBrowser comparison and difference algorithm is particularly designed for C/C++ source code and ignore white spaces and modifications in comments.
15.1 Reference Database
The reference database is the base instrumentation database which are used for the comparison.
To select it click on "Tools->Compare with…" and select a .csmes database.
Switching the working database with the reference database can be performed by clicking on "Tools->Switch databases".
Once the reference file loaded, additional filter possibilities are available in the "Source Browser" and "Method Browser". This filters permits to show/hide, modified, new, deleted or identical procedures and source files.
15.2 Coverage analysis of modified/unmodified source code
CoverageBrowser is able to limit the code coverage analysis to the modified (resp. unmodified) functions.
When selecting the coverage analysis on the modified (resp. unmodified) functions only,
CoverageBrowser treat all unmodified (resp. modified) functions as if they are not instrumented.
Limiting the code coverage analysis to modified functions can be a practical way to verify the new features are tested and to identify the list of tests which have are impacted by a modification.
To limit the code coverage to modified function (resp. unmodified functions) click on "Tools->Analysis on Modified Methods" (resp. "Tools->Analysis on Identical Methods").
Chapter 16 Instrumentation Database
16.1 Merging Instrumentations
Clicking on the menu entry "File->Merge with…"
permits to import the executions, the source code and the instrumentations from other .csmes files.
Comments and code mark as validated are merged together.
Chapter 17 Exporting Statistics
17.1 Statistics per Source File
Clicking on the menu entry "Tools->Export Statistics per Source File" permits to export code coverage statistics of each source file in CSV format.
The file contains the coverage of all combination of source files and executions selected and the execution status (see figure fig:statistic_csv_fig). Comma and semi-colon can be selected as separator in the CSV file using the "File Type" input field.
17.2 Statistics per Method
Clicking on the menu entry "Tools->Export Statistics per Method" permits to export code coverage statistics of each function and procedure in CSV format.
17.3 HTML/XML Report
Clicking on the menu entry "Tools->Generate Report…" permits to export code coverage statistics (per methods, source files, executions, …) of the selected executions. It permits also to list the manually validated and unexecuted code parts.
Chapter 18 Preferences Dialog
18.1 Save/Load Project
- "Save/Restore window position"
- If this option is selected, the position of all windows and toolbars will be restored upon application restart.
- "Reload automatically the last project"
- If this option is selected, the last project opened will automatically be reloaded upon application restart.
- "Saves project automatically on exit"
- Saves the project file automatically without asking on application exit.
18.2 Comments
- "Minimum Comment Size"
- The minimum comment size, is the minimum length requested for a comment.
- "Do not request a comment when setting an item to the ’manually validated’ state"
-
This option is used to allow the user to manually modify
the state of an instrumentation without entering a comment.
Enabling this option should be avoided because modifying the state of an instrumentation should be performed with a valid reason which should be recorded as a comment.
18.3 Watermarks
Watermarks are trigger values that control the background color of:
- the instrumented source files in in the "Source Browser" window.
- the instrumented classes or namespaces in the "Method Browser" window.
- the instrumented functions, methods or procedures in the "Method Browser" window.
Description:
- "Medium/High Coverage Level"
- If the statistic is above this value, the background color is set to green. Otherwise, the color is orange.
- "Low/Medium Coverage Level"
- If the statistic is below this value, the background color is set to red. Otherwise, the color is orange.
18.4 Cache
Description:
- "Execution"
- Maximum number of executions loaded into the RAM.
- "Source"
- Maximum number of source files loaded into the RAM.
| ©2006-2010 |  info@testcocoon.org |