Part II
CoverageBrowser Reference Manual
CoverageBrowser is the graphical interface which enables the user to analyze
test coverage.
The typical usage of coverage browser consists in:
-
Loading the instrumentation database ("
.csmes" file)
which is generated by CoverageScanner.
- Loading the execution report ("
.csmes"’ file).
CoverageBrowser displays them in a tree view and enables the user to individually select
or deselect them for coverage analysis.
- Searching for untested code segments and executing a test.
- Marking dead code or code which can not be tested, as "manually validated".
- Commenting instrumented code areas.
CoverageBrowser saves all data (execution reports, comments, …)
to the instrumentation database.
Chapter 4 Black box and white box testing
CoverageBrowser can be used for white box and black box testing: if no source
code information is available in the instrumentation database (.csmes file),
CoverageBrowser switch in a lightweight version. In this mode, it is not possible to have access to
the source code or to the symbol information, but the user can import and manage its executions.
A black box instrumentation database can generated by clinging on "File->Generate Black Box Configuration".
This instrumentation can be in a second step merged in a white box configuration again.
CoverageBrowser view when performing black box tests
CoverageBrowser view when performing black box tests
|
Chapter 5 Execution Management Window
5.1 Principle
Executions of the instrumented application are managed in a tree view in the "Executions" window.
CoverageBrowser uses a slash ’/’ as separator for grouping measurements together.
For example, tests viewed in figure fig:coveragebrowser_executions_fig
have the following names:
-
SubTest/test1
- SubTest/test2 (12)
- SubTest/test2 (2)
Executions View
Executions View
|
The check-box next to each item can be used to select executions.
The input field
permits to filter the output using regular expressions (see chap. sec:coveragebrowser-filter) and the button button
and
permits to select/unselect
all executions visible. Additional filter which concerns the execution state and the comments can be set using the button "...".
The item text which can be filtered is the full name of the execution. (example: SubTest/test1)
The "Source Browser", "Method Browser" and the "Source Viewer" windows display only the code coverage status
of the selected executions. The button
permits to switch into the test benefit analysis mode (see chap. test-benefit).
The user can set the state of the executed test, which can be:
-
"Unknown"
-
Default state.
- "Passed"
-
This state is used to mark the test as passed (background colour is green)
- "Failed"
-
This state is used to mark the test as failed (background colour is red)
- "Need manual check"
-
This state is used to show the test has to be checked manually (background colour is orange)
Using the context menu or the appropriate docking window, it is possible to comment, rename, delete or merge1 executions together.
CoverageBrowser provides the possibility to use regular expressions for this manipulations. The syntax of it is described in the chapter sec:coveragebrowser-filter and a preview of the operation is calculated automatically.
Here some examples of deleting execution:
-
To delete all execution using the wildcard syntax:
- To delete all executions in
TESTS using the wildcard syntax:
- To delete all executions in
TESTS using the regular expression syntax:
Here some examples of renaming execution:
-
To put all executions in the directory
TESTS set:
| Actual Execution Name | =.* |
| New Execution Name | TESTS/& |
- To move all executions in the directory
TESTS in the directory OLD set:
| Actual Execution Name | =TESTS/(.*) |
| New Execution Name | OLD/\1 |
- To rename all executions in all directories in
testname [directory] set:
| Actual Execution Name | =([^/]*)/([^/]*) |
| New Execution Name | \2 [\1] |
Renaming using Regular Expressions
Renaming using Regular Expressions
|
The name of the test item and also its state can be defined by an external test suite.(see chap.
test-suite-adaptation)
5.2 Loading an Execution Report
The execution report is produced upon application exit.
Its name is defined by the initialization function
__coveragescanner_install of the CoverageScanner library
(see chap. coveragescanner_install).
Its extension is followed by ".csexe".
It contains the list of all executed code segments for each application run.
The execution report is never overwritten but
execution contents are appended.
To load an execution report click on "File->Load Execution Report"
or the icon
on the toolbar.
The dialog as shown in fig:coveragebrowser_loadexec_fig will appear.
Loading Execution Report Dialog
Loading Execution Report Dialog
|
The file can be loaded directly or using a script.
To load directly, click on "file",
enter the path of the ".csexe" file to load
to the free form input box or use the browse button.
If the option "Delete after transferring" is selected, the execution report file
will be deleted after transfer.
The option "When file becomes modified" permits to reopen this dialog
automatically when the execution report is modified.
The "Name" field allows the user to specify the name of the imported instrumentation
if this is not specified in the execution report (see chap. executionreportname). Also, it is
possible to set the execution status (passed, failed or to be checked manually) of all imported executions.
If more than one instrumentation is imported an index is appended to this default name.
The option "Import Preprocessing" permits to select the behaviour
in the case of conflicts or redundant executions:
-
"Ignore Duplicate Executions"
-
Executions which have executed the same code than an execution which are already
imported are ignored.
- "Import Duplicate Executions"
-
Executions are imported if at least one instrumented source code line is executed.
- "Import Duplicate And Empty Executions"
-
All executions are imported.
- "Merge all executions with the same name together"
-
All executions with the same name are merged (i.e. the execution count of each instrumentation is added).
Invalid executions are not imported and
a summary shows when the operation is completed (see figure fig:coveragebrowser_loadreport)
when "Display import summary" option is selected.
Loading Executions - Summary
Loading Executions - Summary
|
If the execution report is not accessible through the file system,
a script can be used. The script has only to print the contain of
the execution report to the standard output (stdout). The standard error
output (stderr) is displayed on the screen and can be used for debugging
purpose. On success, the script must exit with the value 0.
5.3 Test Benefit Analysis Mode
The test benefit analysis mode if activated by clicking on the button
.
In this mode, CoverageBrowser does not display the coverage of a set of tests but the lines
covered by an execution which are not covered by a set of reference executions.
In other words, CoverageBrowser shows what an execution is covering more than a set of other executions.
The list of reference executions are the executions checked in the execution list.
Double-clicking on the name of the execution permits to select the test to analyse.
Only the instrumented lines which are executed by this execution are shown, the other are in the state "hidden".
Also the coverage statistics displayed in the source list are only containing the percentage of
instrumented statements which are only executed by this the selected execution.
If the execution to analyse is present in the list of reference executions,
this one if implicitly remove from the list. (In other words, if execution A
is compared to execution A and B,
CoverageBrowser make a comparison of execution A with B only.)
Comparing the benefit of an execution with itself will in fact not give usefully information.
Test Benefit Analysis Mode
Test Benefit Analysis Mode
|
Chapter 6 Source Browser Window
This feature is not available for black box testing.
The "Source Browser" window can be displayed by clicking
on "View->Source Browser".
Source Browser Window
Source Browser Window
|
Each item is a C/C++ source file and the sub-entries are the list
of included headers which have been instrumented. When an item
the "Source Window" will be displayed.
The "Source Browser" window displays rudimentary code coverage statistics
for each source code file. The color of each item is selected according to code coverage
statistics for each file and the watermarks.(see chap. watermarks)
The input field
permits to filter the output using regular expressions
(see chap. sec:coveragebrowser-filter).
The item text which can be filtered is the full path of the source file. (example: c:\directory\file.cpp)
| Icon | Shortcut | Description |
 | Ctrl+Shift+F | Previous source file |
 | Ctrl+F | Next source file |
Source Browser - Shortcuts
|
This feature is not available for black box testing.
Chapter 7 Method Browser Window
The "Method Browser" window can be opened by clicking
on "View->Method Browser".
Method Browser Window
Method Browser Window
|
The "Method Browser" window displays a the code coverage
statistics of each C/C++ functions, classes and namespaces.
Clicking on a function permits to show all instrumented lines of it
in the "Source Viewer".
The input field
permits to filter the output using regular expressions
(see chap. sec:coveragebrowser-filter).
The item text which can be filtered is the symbol name including the class name and the namespace.
(example: MyNamespace::MyClass::MyProc)
This feature is not available for black box testing.
Chapter 8 Source Viewer Window
8.1 Source Display
The "Source Viewer" window can be displayed by clicking on "View->New Source Window".
Source Window
Source Window
|
The "Source Viewer" window displays the source file or its C/C++ preprocessed view.
Clicking on
enables the user to toggle between the 2 different views.
The source code is colored with code coverage instrumentations.
The colors used are described in section sec:color_convension.
By selecting an area with the mouse, corresponding instrumentations are highlighted and a
detailed description of them is displayed in the "Explanation"
window (see chap. code-coverage-explanation). It is possible to navigate between
instrumentations using the navigation buttons
and
.
Navigation buttons in yellow, blue, red and green permit to jump to the next or previous
comments, manually validated instrumentations, non-executed code parts or executed code parts.
Clicking on the source code selects the nearest instrumentation.
If a comment is entered for an instrumentation, the icon
is displayed in the margin.
On the right side, CoverageBrowser displays the test coverage
count1
or the code coverage count2
for each line.
If a source code line contains more than one instrumentation,
CoverageBrowser display the range of their counts.
| Mouse Wheel | Description |
| Wheel | Scroll up/down |
| Ctrl+Wheel | Zoom in/out |
| Shift+Wheel | Next/previous instrumentation |
Source Display - Mouse Wheel
|
This feature is not available for black box testing.
8.2 Color Convention
Instrumentations are displayed in a source window using different colors:
-
Green - "Executed"
-
An instrumentation is displayed in green when the code has been executed.
- Orange - "Partially Executed"
-
An instrumentation is marked as "Partially Executed" when it
is not completely executed. This occurs when
a Boolean expression was only true or false for example.
In the case of a source code line which contains more than one instrumentation,
the line is marked as "Partially Executed" when one of its instrumentations
has not been "Executed".
A detailed information is displayed in the
"Explanation" window (see chap. code-coverage-explanation).
- Red - "Never Executed" or "Execution count too low"
-
An instrumentation is displayed in red when the code is never executed
or when the execution count is lower that than the execution count requested.
- Blue - "Manually Set To Be Executed"
-
The user has the possibility to mark an instrumentation as ’Manually Validated’.
This is usually to exclude dead code or code which
cannot be tested for code coverage statistics.
This state is only relevant if executions are in a
"Never Executed" or "Partially Executed" state.
- Gray - "Unknown" or "Hidden"
-
Gray is used when no information about instrumentation is available.
This occurs when no executions are selected or when analysing the benefit
of tests (see chap. test-benefit).
- Magenta - "Registration Needed"
-
Instrumentations displayed in magenta require product registration to be visible.
The evaluation version only displays a subset of the instrumentations.
See chapter order to register TestCocoon.
8.3 Comments
8.3.1 Editing Comments
It is possible to add a comment by selecting an instrumentation and clicking on
the context menu entry "Add/Edit Comment", the main menu entry
"instrumentation->Add/Edit Comment" or the icon
on the toolbar.
The "Comment" Window fig:coveragebrowser_comment_edt appears and allows a comment to be edited.
The most recently entered comments can be retrieved by clicking on the "Last Comments" selection field.
Basic text formatting is possible using the integrated toolbar buttons (see comments_shortcut).
Comment Editing
Comment Editing
|
If a minimal length for a comment is set, the comment can only
be entered if this is reached (see chap.
minimum_comment_size).
The comment is printed in the explanation in a yellow box and the icon (
)
is displayed in the source window near the line number.
This feature is not available for black box testing.
8.3.2 Removing Comments
It is possible to remove a comment by selecting an instrumentation
and clicking on the context menu entry "Remove Comment", the main menu
entry "instrumentation->Remove Comment" or the icon
on the toolbar.
This feature is not available for black box testing.
Chapter 9 Code Coverage Explanation Window
The "Explanation" Window fig:coveragebrowser_expl_fig
is a docking window which is automatically updated with a detailed
description of the selected instrumentations of the source window.
For each instrumentation, the following information is displayed:
-
A short description of the instrumentation state (see chap. sec:color_convension).
- The preprocessed source code which is concerned by the instrumentation.
- For Boolean expressions, the truth-table which shows executed and unexecuted states.
- The list of executions which are executing the portion of code.
- User comments.
Explanation Window
Explanation Window
|
CoverageBrowser displays the truth-table in the case of a Boolean expression which is partially executed.
The truth-table indicates which value the expression has or has not reached during execution.
Example: the truth-table truth_table indicates that the expression was false but not true.
This feature is not available for black box testing.
The "Statistic" Window fig:coveragebrowser_statistics
is a docking window which is automatically updated with the
code coverage statistic for the whole project.
If the coverage level is greater than one, the "Statistic" Window displays
the statistics of the current level and the level one.
Statistic Window
Statistic Window
|
Chapter 11 Filter using wildcard expression or regular expressions
CoverageBrowser provides a generic filtering mechanism of rows using wildcard or regular expressions.
Wildcard expressions are activated per default and regular expressions are selected when the expression starts with
an equal sign (’=’).
Clicking on the filter icon converts the expression from wildcard into regular form as far as this is possible
and vice versa.
11.1 Wildcard Expression
| Element | Meaning |
* | any characters (0 or more) |
? | any character |
[...] | set of character |
|
Example: foo*bar match any tests containing the string foo followed by bar.
11.2 Regular Expression
The first character must be ’=’ to activate the regular expressions.
11.2.1 Pattern matching
| Element | Meaning |
c | Any character represents itself unless it has a special regexp
meaning. Thus c matches the character c. |
\c | A character that follows a backslash matches the character itself
except where mentioned below. For example if you wished to match
a literal caret at the beginning of a string you would write \^. |
\a | This matches the ASCII bell character (BEL, 0x07). |
\f | This matches the ASCII form feed character (FF, 0x0C). |
\n | This matches the ASCII line feed character (LF, 0x0A, Unix newline). |
\r | This matches the ASCII carriage return character (CR, 0x0D). |
\t | This matches the ASCII horizontal tab character (HT, 0x09). |
\v | This matches the ASCII vertical tab character (VT, 0x0B). |
\xhhhh | This matches the Unicode character corresponding to the hexadecimal
number hhhh (between 0x0000 and 0xFFFF). |
\0ooo (i.e., zero ooo) | matches the ASCII/Latin1 character corresponding to the
octal number ooo (between 0 and 0377). |
. (dot) | This matches any character (including newline). |
\d | This matches a digit. |
\D | This matches a non-digit. |
\s | This matches a whitespace. |
\S | This matches a non-whitespace. |
\w | This matches a word character. |
\W | This matches a non-word character. |
^ | The caret negates the character set if it occurs as the first character, i.e.
immediately after the opening square bracket. For example, [abc] matches
'a' or 'b' or 'c', but [^abc] matches anything
except 'a' or 'b' or 'c'. |
- | The dash is used to indicate a range of characters, for example [W-Z]
matches 'W' or 'X' or 'Y' or 'Z'. |
E? | Matches zero or one occurrence of E. This quantifier means "the previous expression
is optional" since it will match whether or not the expression occurs in the string.
It is the same as E{0,1}. For example dents? will match 'dent'
and 'dents'. |
E+ | Matches one or more occurrences of E. This is the same as E{1,}. For example,
0+ will match '0', '00', '000', etc… |
E* | Matches zero or more occurrences of E. This is the same as E{0,}. The *
quantifier is often used by a mistake. Since it matches zero or more occurrences it
will match no occurrences at all. For example if we want to match strings that end
in whitespace and use the regexp \s*$ we would get a match on every string.
This is because we have said find zero or more whitespace followed by the end of
string, so even strings that don’t end in whitespace will match. The regexp we want
in this case is \s+$ to match strings that have at least one whitespace at the end. |
E{n} | Matches exactly n occurrences of the expression. This is the same as repeating the
expression n times. For example, x{5} is the same as xxxxx. It is also
the same as E{n,n}, e.g. x{5,5}. |
E{n,} | Matches at least n occurrences of the expression. |
E{,m} | Matches at most m occurrences of the expression. This is the same as E{0,m}. |
E{n,m} | Matches at least n occurrences of the expression and at most m
occurrences of the expression. |
() | Permits to group expressions into sub-expressions. |
| | Alternative.
Example: "aaa|bbb" matches the string "aaa" or "bbb". |
|
11.2.2 String substitution
| Element | Meaning |
& | Matched expression |
\n | sub-expression number n.
Example: the regular expression is
"(.*):([0-9]*)" matches the string "joe:18".
The replacement string "\1 is \2" will produce the result: "joe is 18" |
|
Chapter 12 Code/Test Coverage Level
The menu entry "Instrumentation->Level:x" permits to set the targeted code
coverage count or, if the compiled with instrumentation hit
support1, the targeted test coverage count.
The level is corresponding of the number of code/test coverage count necessary
to consider that an instrumented code is executed.
Example: Setting the level to 10, will made necessary to execute 10 times
the each line of the source code if compiled with code coverage count.
If compiled with code coverage hit, 10 execution runs need to execute
each lines of the source code.
The menu entry "Tools->Test Coverage Count Mode" and the button
permits to switch between code coverage count and test coverage count analysis. This simulates the behaviour of the compilation with code coverage hit support2 when the project is compiled with code coverage count support3.
Chapter 13 Code Coverage Algorithm
CoverageBrowser displays the code coverage analysis (branch, decision or condition) generated be CoverageScanner.
But "Instrumentation->Coverage Method->Branch only" permits to reduce the analysis to the code coverage of branches.
This produces the same result as compiling with the --cs-branch of CoverageScanner.
"Instrumentation->Coverage Method->Decision, Condition and Branches" permits to show the code coverage analysis at the level defined at the compilation.
Here a short overview of the command line options necessary for each code coverage analysis method:
| Coverage analysis | CoverageScanner command line option |
| Branch | --cs-branch |
| Decision with full instrumentation | --cs-decision --cs-full-instrumentation |
| Decision with partial instrumentation | --cs-decision |
| Condition with full instrumentation | --cs-full-instrumentation |
| Condition with partial instrumentation | (default) |
|
Chapter 14 Optimized Execution Order
CoverageBrowser is able to calculate an optimized order of the executions
(i.e.: the order of tests which permits to maximize the code coverage after
each execution).
This order is specially adapted to manual testing: following this order permits
to execute first the tests which are giving a high code coverage and so detecting rapidly
errors in the first test executions.
To calculate the execution order proceed as follows:
-
Select a set of executions in the "Executions" window.
- Click on "Tools->Optimized Execution Order…".
The window fig:optimized_execution_order_fig will de display.
Optimized Execution Order
Optimized Execution Order
|
Chapter 15 Comparing Code Coverage of Two Software Releases
CoverageBrowser is able to instrumentation database together in order to:
-
check is the modified/unmodified code is correctly tested.
- find which tests are impacted by a source code modification.
This feature is particularly adapted to compare two releases together which contains small modifications (bug fixes only)
and to limit the tests of the modified code only.
In this mode CoverageBrowser uses the following typographic rules:
| Rule | Source Window | Method List | Source List |
| Normal font | Identical1 source part | Identical1 methods | Identical1 files |
| Bold | | Modified methods | Modified files |
| Bold+Underline | New text inserted | New methods | New files |
Bold+Strike | Deleted text | Deleted methods | Deleted files |
|
CoverageBrowser comparison and difference algorithm is particularly designed for C/C++ source code
and ignore white spaces and modifications in comments.
15.1 Reference Database
The reference database is the base instrumentation database which are used for the comparison.
To select it click on "Tools->Compare with…" and select a .csmes database.
Switching the working database with the reference database can be performed by clicking on "Tools->Switch databases".
Once the reference file loaded, additional filter possibilities are available in the "Source Browser" and "Method Browser".
This filters permits to show/hide, modified, new, deleted or identical procedures and source files.
15.2 Coverage analysis of modified/unmodified source code
CoverageBrowser is able to limit the code coverage analysis to the modified (resp. unmodified) functions.
When selecting the coverage analysis on the modified (resp. unmodified) functions only,
CoverageBrowser treat all unmodified (resp. modified) functions as if they are not instrumented.
Limiting the code coverage analysis to modified functions can be a practical way to verify the new features are tested and to identify the list of tests which have are impacted by a modification.
To limit the code coverage to modified function (resp. unmodified functions) click on "Tools->Analysis on Modified Methods" (resp. "Tools->Analysis on Identical Methods").
Chapter 16 Instrumentation Database
16.1 Merging Instrumentations
Clicking on the menu entry "File->Merge with…"
permits to import the executions, the source code and the instrumentations from other .csmes files.
Comments and code mark as validated are merged together.
Chapter 17 Exporting Statistics
17.1 Statistics per Source File
Clicking on the menu entry "Tools->Export Statistics per Source File"
permits to export code coverage statistics of each source file in CSV format.
The file contains the coverage of all combination of source files and executions selected
and the execution status (see figure fig:statistic_csv_fig). Comma and semi-colon can be selected as separator
in the CSV file using the "File Type" input field.
Statistics Format - CSV table
Statistics Format - CSV table
|
17.2 Statistics per Method
Clicking on the menu entry "Tools->Export Statistics per Method"
permits to export code coverage statistics of each function and procedure in
CSV format.
17.3 HTML/XML Report
Clicking on the menu entry "Tools->Generate Report…"
permits to export code coverage statistics (per methods, source files, executions, …)
of the selected executions. It permits also to list the manually validated and unexecuted code parts.
This feature is not available for black box testing.
18.1 Save/Load Project
-
"Save/Restore window position"
-
If this option is selected, the position of
all windows and toolbars will be restored
upon application restart.
- "Reload automatically the last project"
-
If this option is selected, the last
project opened will automatically be reloaded
upon application restart.
- "Saves project automatically on exit"
-
Saves the project file automatically without asking on application exit.
18.2 Comments
-
"Minimum Comment Size"
-
The minimum comment size, is the minimum length requested
for a comment.
- "Do not request a comment when setting an item to the ’manually validated’ state"
-
This option is used to allow the user to manually modify
the state of an instrumentation without entering a comment.
Enabling this option should be avoided because modifying the
state of an instrumentation should be performed with
a valid reason which should be recorded as a comment.
18.3 Watermarks
Watermarks are trigger values that control the background color of:
-
the instrumented source files in in the "Source Browser" window.
- the instrumented classes or namespaces in the "Method Browser" window.
- the instrumented functions, methods or procedures in the "Method Browser" window.
Description:
-
"Medium/High Coverage Level"
-
If the statistic is above this value, the background color is set to green.
Otherwise, the color is orange.
- "Low/Medium Coverage Level"
-
If the statistic is below this value, the background color is set to red.
Otherwise, the color is orange.
Description:
-
"Execution"
-
Maximum number of executions loaded into the RAM.
- "Source"
-
Maximum number of source files loaded into the RAM.