Code Coverage Measurement for C/C++
Part II |
- Introduction
- Black box and white box testing
- Execution Management Window
- Source Browser Window
- Method Browser Window
- Source Viewer Window
- Explanation Window
- Statistic Window
- Filter
- Code Coverage Level
- Code Coverage Algorithm
- Instrumentation Database
- Exporting Statistics
- Options
Chapter 3 Introduction
CoverageBrowser is the graphical interface which enables the user to analyze test coverage.
The typical usage of coverage browser consists in:
-
Loading the instrumentation database ("
.csmes" file) which is generated by CoverageScanner. - Loading the execution report ("
.csmes"’ file). CoverageBrowser displays them in a tree view and enables the user to individually select or deselect them for coverage analysis. - Searching for untested code segments and executing a test.
- Marking dead code or code which can not be tested, as "manually validated".
- Commenting instrumented code areas.
CoverageBrowser saves all data (execution reports, comments, …) to the instrumentation database.
Chapter 4 Black box and white box testing
CoverageBrowser can be used for white box and black box testing: if no source
code information is available in the instrumentation database (.csmes file),
CoverageBrowser switch in a lightweight version. In this mode, it is not possible to have access to
the source code or to the symbol information, but the user can import and manage its executions.
A black box instrumentation database can generated by clinging on "File->Generate Black Box Configuration".
This instrumentation can be in a second step merged in a white box configuration again.
Chapter 5 Execution Management Window
5.1 Principle
Executions of the instrumented application are managed in a tree view in the "Executions" window.
CoverageBrowser uses a slash ’/’ as separator for grouping measurements together.
For example, tests viewed in figure 5.1
have the following names:
- SubTest/test1
- SubTest/test2 (12)
- SubTest/test2 (2)
The check-box next to each item can be used to select executions.
The input field 
permits to filter the output using regular expressions
(see chap. 11)
and the button button 
and 
permits to select/unselect
all executions visible.
The item text which can be filtered is the full name of the execution. (example: SubTest/test1)
The "Source Browser", "Method Browser" and the "Source Viewer" windows display only the code coverage status
of the selected executions. The button 
permits to switch into the test benefit analysis mode
(see chap. 5.3)
.
The user can set the state of the executed test, which can be:
- "Unknown"
- Default state.
- "Passed"
- This state is used to mark the test as passed (background colour is green)
- "Failed"
- This state is used to mark the test as failed (background colour is red)
- "Need manual check"
- This state is used to show the test has to be checked manually (background colour is orange)
Using the context menu or the appropriate docking window, it is possible to comment, rename, delete or merge1 executions together.
CoverageBrowser provides the possibility to use regular expressions for this manipulations. The syntax of it is described in the chapter 11 and a preview of the operation is calculated automatically.
Here some examples of deleting execution:
-
To delete all execution using the wildcard syntax:
Execution Name * - To delete all executions in
TESTSusing the wildcard syntax:Execution Name TESTS/* - To delete all executions in
TESTSusing the regular expression syntax:Execution Name =TESTS/.*
Here some examples of renaming execution:
-
To put all executions in the directory
TESTSset:Actual Execution Name =.*New Execution Name TESTS/& - To move all executions in the directory
TESTSin the directoryOLDset:Actual Execution Name =TESTS/(.*)New Execution Name OLD/\1 - To rename all executions in all directories in
testname [directory]set:Actual Execution Name =([^/]*)/([^/]*)New Execution Name \2 [\1]
5.2 Loading an Execution Report
The execution report is produced upon application exit.
Its name is defined by the initialization function
__coveragescanner_install of the CoverageScanner library
(see chap. 22.3.1)
.
Its extension is followed by ".csexe".
It contains the list of all executed code segments for each application run.
The execution report is never overwritten but
execution contents are appended.
To load an execution report click on "File->Load Execution Report"
or the icon 
on the toolbar.
The dialog as shown in 5.3 will appear.
The file can be loaded directly or using a script.
To load directly, click on "file",
enter the path of the ".csexe" file to load
to the free form input box or use the browse button.
If the option "Delete after transferring" is selected, the execution report file
will be deleted after transfer.
The option "When file becomes modified" permits to reopen this dialog
automatically when the execution report is modified.
The "Name" field allows the user to create the name of the imported instrumentation
if this is not specified in the execution report
(see chap. 22.3.2)
.
If more than one instrumentation is imported an index is appended to this default name.
The option "Import Preprocessing" permits to select the behaviour
in the case of conflicts or redundant executions:
- "Ignore Duplicate Executions"
- Executions which have executed the same code than an execution which are already imported are ignored.
- "Import Duplicate Executions"
- Executions are imported if at least one instrumented source code line is executed.
- "Import Duplicate And Empty Executions"
- All executions are imported.
- "Merge all executions with the same name together"
- All executions with the same name are merged (i.e. the execution count of each instrumentation is added).
Invalid executions are not imported and
a summary shows when the operation is completed (see figure 5.2)
when "Display import summary" option is selected.
If the execution report is not accessible through the file system,
a script can be used. The script has only to print the contain of
the execution report to the standard output (stdout). The standard error
output (stderr) is displayed on the screen and can be used for debugging
purpose. On success, the script must exit with the value 0.
5.3 Test Benefit Analysis Mode
The test benefit analysis mode if activated by clicking on the button .
In this mode, CoverageBrowser does not display the coverage of a set of tests but the lines
covered by an execution which are not covered by a set of reference executions.
In other words, CoverageBrowser shows what an execution is covering more than a set of other executions.
The list of reference executions are the executions checked in the execution list.
Clicking2
on the name of the execution permits to select the test to analyse.
Only the instrumented lines which are executed by this execution are shown, the other are in the state "hidden".
Also the coverage statistics displayed in the source list are only containing the percentage of
instrumented statements which are only executed by this the selected execution.
Chapter 6 Source Browser Window
The "Source Browser" window can be displayed by clicking
on "View->Source Browser".
Each item is a C/C++ source file and the sub-entries are the list
of included headers which have been instrumented. When an item
the "Source Window" will be displayed.
The "Source Browser" window displays rudimentary code coverage statistics
for each source code file. The color of each item is selected according to code coverage
statistics for each file and the watermarks.
(see chap. 16.3)
The input field permits to filter the output using regular expressions
(see chap. 11)
.
The item text which can be filtered is the full path of the source file. (example: c:\directory\file.cpp)
Icon Shortcut Description Ctrl+Shift+P Previous source file Ctrl+Shift+N Next source file
Table 6.1: Source Browser - Shortcuts
Chapter 7 Method Browser Window
The "Method Browser" window can be opened by clicking
on "View->Method Browser".
The "Method Browser" window displays a the code coverage
statistics of each C/C++ functions, classes and namespaces.
Clicking on a function permits to show all instrumented lines of it
in the "Source Viewer".
The input field permits to filter the output using regular expressions
(see chap. 11)
.
The item text which can be filtered is the symbol name including the class name and the namespace.
(example: MyNamespace::MyClass::MyProc)
Chapter 8 Source Viewer Window
8.1 Source Display
The "Source Viewer" window can be displayed by clicking on "View->New Source Window".
The "Source Viewer" window displays the source file or its C/C++ preprocessed view.
Clicking on 
enables the user to toggle between the 2 different views.
The source code is colored with code coverage instrumentations.
The colors used are described in section 8.2.
By selecting an area with the mouse, corresponding instrumentations are highlighted and a
detailed description of them is displayed in the "Explanation"
window
(see chap. 9)
. It is possible to navigate between
instrumentations using the navigation buttons 
and 
or by clicking on the source code.
This will select the nearest instrumentation.
The combo box from the toolbar makes it possible to configure filtering via a line folding mechanism.
The folding marks + and - are displayed in
the margin and hide/show lines according to their status.
The status for the folding marks can be:
- "All"
- All instrumentations are selected.
- "Tested"
- Only instrumentations which are fully executed are selected.
- "Untested"
- Instrumentations which are not or only partially executed are selected.
- "Manually Validated"
- Only instrumentations which are manually set to the "executed" state are selected.
Navigation buttons have the following settings.
If a comment is entered for an instrumentation, the icon 
is displayed in the margin.
On the right side, CoverageBrowser displays the test coverage
count1
or the code coverage count2
for each line.
If a source code line contains more than one instrumentation,
CoverageBrowser display the range of their counts.
Mouse Wheel Description Wheel Scroll up/down Ctrl+Wheel Zoom in/out Shift+Wheel Next/previous instrumentation
Table 8.1: Source Display - Mouse Wheel
Table 8.2: Source Display - Shortcuts
8.2 Color Convention
Instrumentations are displayed in a source window using different colors:
- Green - "Executed"
- An instrumentation is displayed in green when the code has been executed.
- Orange - "Partially Executed"
- An instrumentation is marked as "Partially Executed" when it is not completely executed. This occurs when a Boolean expression was only true or false for example. In the case of a source code line which contains more than one instrumentation, the line is marked as "Partially Executed" when one of its instrumentations has not been "Executed". A detailed information is displayed in the "Explanation" window (see chap. 9) .
- Red - "Never Executed" or "Execution count too low"
- An instrumentation is displayed in red when the code is never executed or when the execution count is lower that than the execution count requested.
- Blue - "Manually Set To Be Executed"
- The user has the possibility to mark an instrumentation as ’Manually Validated’. This is usually to exclude dead code or code which cannot be tested for code coverage statistics. This state is only relevant if executions are in a "Never Executed" or "Partially Executed" state.
- Gray - "Unknown" or "Hidden"
- Gray is used when no information about instrumentation is available. This occurs when no executions are selected or when analysing the benefit of tests (see chap. 5.3) .
- Magenta - "Registration Needed"
- Instrumentations displayed in magenta require product registration to be visible. The evaluation version only displays a subset of the instrumentations. See chapter C.1 to register CoverageMeter.
8.3 Comments
8.3.1 Editing Comments
It is possible to add a comment by selecting an instrumentation and clicking on
the context menu entry "Add/Edit Comment", the main menu entry
"instrumentation->Add/Edit Comment" or the icon on the toolbar.
The "Comment" Window 8.3.1 appears and allows a comment to be edited. The most recently entered comments can be retrieved by clicking on the "Last Comments" selection field. Basic text formatting is possible using the integrated toolbar buttons (see 8.3).
The comment is printed in the explanation in a yellow box and the icon ()
is displayed in the source window near the line number.
Icon Shortcut Description Ctrl+B Bold Ctrl+I Italic Ctrl+U Underline Ctrl+J Justify Ctrl+R Align Right Ctrl+L Align Left Ctrl+M Center Ctrl+Z Undo Ctrl+Shift+Z Redo
Table 8.3: Comments - Shortcuts
8.3.2 Removing Comments
It is possible to remove a comment by selecting an instrumentation
and clicking on the context menu entry "Remove Comment", the main menu
entry "instrumentation->Remove Comment" or the icon 
on the toolbar.
Chapter 9 Code Coverage Explanation Window
The "Explanation" Window 9.1
is a docking window which is automatically updated with a detailed
description of the selected instrumentations of the source window.
For each instrumentation, the following information is displayed:
- A short description of the instrumentation state (see chap. ??) .
- The preprocessed source code which is concerned by the instrumentation.
- For Boolean expressions, the truth-table which shows executed and unexecuted states.
- The list of executions which are executing the portion of code.
- User comments.
CoverageBrowser displays the truth-table in the case of a Boolean expression which is partially executed.
The truth-table indicates which value the expression has or has not reached during execution.
Example: the truth-table 9.1 indicates that the expression was false but not true.
TRUE FALSE no yes
Table 9.1: Truth-Table Example
Chapter 10 Statistic Window
The "Statistic" Window 10.1
is a docking window which is automatically updated with the
code coverage statistic for the whole project.
If the coverage level is greater than one, the "Statistic" Windows displays
the statistics of the current level and the level one.
Chapter 11 Filter using wildcard expression or regular expressions
CoverageBrowser provides a generic filtering mechanism of rows using wildcard or regular expressions.
Wildcard expressions are activated per default and regular expressions are selected when the expression starts with
an equal sign (’=’).
Clicking on the filter icon converts the expression from wildcard into regular form as far as this is possible
and vice versa.
Icon Description The filter uses regular expression syntax. The filter uses wildcard syntax. Syntax error. More information are displayed in the status bar.
Table 11.1: Filter States
11.1 Wildcard Expression
| Element | Meaning |
* | any characters (0 or more) |
? | any character |
[...] | set of character |
Example: foo*bar match any tests containing the string foo followed by bar.
11.2 Regular Expression
The first character must be ’=’ to activate the regular expressions.
11.2.1 Pattern matching
| Element | Meaning |
c | Any character represents itself unless it has a special regexp
meaning. Thus c matches the character c. |
\c | A character that follows a backslash matches the character itself
except where mentioned below. For example if you wished to match
a literal caret at the beginning of a string you would write \^. |
\a | This matches the ASCII bell character (BEL, 0x07). |
\f | This matches the ASCII form feed character (FF, 0x0C). |
\n | This matches the ASCII line feed character (LF, 0x0A, Unix newline). |
\r | This matches the ASCII carriage return character (CR, 0x0D). |
\t | This matches the ASCII horizontal tab character (HT, 0x09). |
\v | This matches the ASCII vertical tab character (VT, 0x0B). |
\xhhhh | This matches the Unicode character corresponding to the hexadecimal
number hhhh (between 0x0000 and 0xFFFF). |
\0ooo (i.e., zero ooo) | matches the ASCII/Latin1 character corresponding to the
octal number ooo (between 0 and 0377). |
. (dot) | This matches any character (including newline). |
\d | This matches a digit. |
\D | This matches a non-digit. |
\s | This matches a whitespace. |
\S | This matches a non-whitespace. |
\w | This matches a word character. |
\W | This matches a non-word character. |
^ | The caret negates the character set if it occurs as the first character, i.e.
immediately after the opening square bracket. For example, [abc] matches
'a' or 'b' or 'c', but [^abc] matches anything
except 'a' or 'b' or 'c'. |
- | The dash is used to indicate a range of characters, for example [W-Z]
matches 'W' or 'X' or 'Y' or 'Z'. |
E? | Matches zero or one occurrence of E. This quantifier means "the previous expression
is optional" since it will match whether or not the expression occurs in the string.
It is the same as E{0,1}. For example dents? will match 'dent'
and 'dents'. |
E+ | Matches one or more occurrences of E. This is the same as E{1,}. For example,
0+ will match '0', '00', '000', etc… |
E* | Matches zero or more occurrences of E. This is the same as E{0,}. The *
quantifier is often used by a mistake. Since it matches zero or more occurrences it
will match no occurrences at all. For example if we want to match strings that end
in whitespace and use the regexp \s*$ we would get a match on every string.
This is because we have said find zero or more whitespace followed by the end of
string, so even strings that don’t end in whitespace will match. The regexp we want
in this case is \s+$ to match strings that have at least one whitespace at the end. |
E{n} | Matches exactly n occurrences of the expression. This is the same as repeating the
expression n times. For example, x{5} is the same as xxxxx. It is also
the same as E{n,n}, e.g. x{5,5}. |
E{n,} | Matches at least n occurrences of the expression. |
E{,m} | Matches at most m occurrences of the expression. This is the same as E{0,m}. |
E{n,m} | Matches at least n occurrences of the expression and at most m
occurrences of the expression. |
() | Permits to group expressions into sub-expressions. |
| | Alternative.
Example: "aaa|bbb" matches the string "aaa" or "bbb". |
11.2.2 String substitution
| Element | Meaning |
& | Matched expression |
\n | sub-expression number n.
Example: the regular expression is
"(.*):([0-9]*)" matches the string "joe:18".
The replacement string "\1 is \2" will produce the result: "joe is 18" |
Chapter 12 Code/Test Coverage Level
The menu entry "Instrumentation->Level:x" permits to set the targeted code coverage count or, if the compiled with instrumentation hit support1, the targeted test coverage count.
The level is corresponding of the number of code/test coverage count necessary
to consider that an instrumented code is executed.
Example: Setting the level to 10, will made necessary to execute 10 times
the each line of the source code if compiled with code coverage count.
If compiled with code coverage hit, 10 execution runs need to execute
each lines of the source code.
The menu entry "Tools->Test Coverage Count Mode" and the button 
permits to switch between code coverage count and test coverage count analysis. This simulates the behaviour of the compilation with code coverage hit support2 when the project is compiled with code coverage count support3.
Chapter 13 Code Coverage Algorithm
CoverageBrowser displays the code coverage analysis (branch, decision or condition) generated be CoverageScanner.
But "Instrumentation->Coverage Method->Branch only" permits to reduce the analysis to the code coverage of branches.
This produces the same result as compiling with the --cs-branch of CoverageScanner.
"Instrumentation->Coverage Method->Decision, Condition and Branches" permits to show the code coverage analysis at the level defined at the compilation.
Here a short overview of the command line options necessary for each code coverage analysis method:
Chapter 14 Instrumentation Database
14.1 Merging Instrumentations
Clicking on the menu entry "File->Merge with…"
permits to import the executions, the source code and the instrumentations from other .csmes files.
Comments and code mark as validated are merged together.
Chapter 15 Exporting Statistics
15.1 Statistics per Source File
Clicking on the menu entry "Tools->Export Statistics per Source File" permits to export code coverage statistics of each source file in CSV format.
The file contains the coverage of all combination of source files and executions selected and the execution status (see figure 15.1). Comma and semi-colon can be selected as separator in the CSV file using the "File Type" input field.
15.2 Statistics per Method
Clicking on the menu entry "Tools->Export Statistics per Method" permits to export code coverage statistics of each function and procedure in CSV format.
15.3 HTML Report
Clicking on the menu entry "Tools->Generate HTML Report" permits to export code coverage statistics (per methods, source files, executions, …) of the selected executions. It permits also to list the manually validated and unexecuted code parts.
Chapter 16 Options Dialog
16.1 Save/Load Project
- "Save/Restore window position"
- If this option is selected, the position of all windows and toolbars will be restored upon application restart.
- "Reload automatically the last project"
- If this option is selected, the last project opened will automatically be reloaded upon application restart.
- "Saves project automatically on exit"
- Saves the project file automatically without asking on application exit.
16.2 Comments
- "Minimum Comment Size"
- The minimum comment size, is the minimum length requested for a comment.
- "Do not request a comment when setting an item to the ’manually validated’ state"
-
This option is used to allow the user to manually modify
the state of an instrumentation without entering a comment.
Enabling this option should be avoided because modifying the state of an instrumentation should be performed with a valid reason which should be recorded as a comment.
16.3 Watermarks
Watermarks are trigger values that control the background color of:
- the instrumented source files in in the "Source Browser" window.
- the instrumented classes or namespaces in the "Method Browser" window.
- the instrumented functions, methods or procedures in the "Method Browser" window.
Description:
- "Medium/High Coverage Level"
- If the statistic is above this value, the background color is set to green. Otherwise, the color is orange.
- "Low/Medium Coverage Level"
- If the statistic is below this value, the background color is set to red. Otherwise, the color is orange.
16.4 Cache
Description:
- "Execution"
- Maximum number of executions loaded into the RAM.
- "Source"
- Maximum number of source files loaded into the RAM.
8 rue Krummenacker, F-67620 Soufflenheim 