Approach 1
PHP code coverage data can be collected using the sebastianbergmann/php-code-coverage composer module. But this is easier when we are running PHP unit test.
When we test our application using a browser, either through manual testing or through automation like Selenium or QTP. Every request get’s generated by the Web Browser and is handled by a Web Server. This in many cases would be either Nginx or Apache.
It is important to know what level of code get’s covered through the QA process. In PHP two approaches are used to serve client requests. First approach where every request gets forwarded to a single PHP file, and the file internally routes the request further. Frameworks like CodeIgniter, Wordpress, Laravel use the first approach. Second approach is where individual files directly serve the PHP request, this usually happens in custom PHP applications.
For us it was important to have code coverage as a configuration rather than a code change. So we needed a way to execute some code before every request.
php.ini configuration
Digging into php.ini documentation revelead two supported variables
auto_prepend_file string - Specifies the name of a file that is automatically parsed before the main file. The file is included as if it was called with the require function, so include_path is used. The special value none disables auto-prepending.
auto_append_file string - Specifies the name of a file that is automatically parsed after the main file. The file is included as if it was called with the require function, so include_path is used. The special value none disables auto-appending.
Request Code Coverage Dumper
Now that we know the two variables auto_prepend_file and auto_append_file, we need a PHP code that gets appended at the start and dumps the code coverage at the end of the request.
My first approach was to use the php-code-coverage module to start coverage at every request and dump the coverage data at the end of the request.
composer.json
{
"require" : {
"phpunit/php-code-coverage":"4.*" }
}
/var/www/codecoverage/start.php
<?php
require_once "vendor/autoload.php";
$current_dir = __DIR__;
$coverage = new SebastianBergmann\CodeCoverage\CodeCoverage;
$filter = $coverage->filter();
$filter->addDirectoryToWhitelist("/var/www/html");
$test_name = (isset($_COOKIE['test_name']) && !empty($_COOKIE['test_name'])) ? $_COOKIE['test_name'] : 'unknown_test_' . time();
$coverage->start($test_name);
function end_coverage()
{
global $test_name;
global $coverage;
global $filter;
global $current_dir;
$coverageName = $current_dir . '/coverages/coverage-' . $test_name . '-' . microtime(true);
try
{
$coverage->stop();
$writer = new \SebastianBergmann\CodeCoverage\Report\Html\Facade;
$writer->process($coverage, $current_dir . '/report/');
$writer = new SebastianBergmann\CodeCoverage\Report\PHP();
$coverageName = $current_dir .'/coverages/coverage-'. $test_name . '-' . microtime(true);
$writer->process($coverage, $coverageName . '.php');
}
catch (Exception $ex)
{
file_put_contents($coverageName . '.ex', $ex);
}
}
class coverage_dumper
{
function __destruct()
{
end_coverage();
}
}
$_coverage_dumper = new coverage_dumper();
The code above is self explainatory. But there are few points I would like to highlight.
Whitelist and Blacklist
$filter->addDirectoryToWhitelist("/var/www/html");
If you don’t add any directory to white list, no code coverage would be generated. First add all the whitelist directories and then all the blacklist directories. If you are using composer in your project, then you would want to black list the vendor directory
Code Coverage to Test Case mapping
$test_name = (isset($_COOKIE['test_name']) && !empty($_COOKIE['test_name'])) ? $_COOKIE['test_name'] : 'unknown_test_' . time();
While dumping the code coverage, a test name can be associated with the same. This is important because a QA test case will have multiple request within the same test. Ability to map coverage to test cases would give more insights. To do so, you can set a cookie in the browser before starting the test case.
This can be done in multiple ways
* Create a dummy page which takes GET parameter and sets the cookie test_name with the given value
* Use Selenium API to add a cookie to the browser
* Use some other approach to pass the test_name, instead of the cookies. This would require update in start.php
Enabling the code coverage collection
There are multiple ways to enable code coverage. We will look at all the configuration
1. Using php.ini
Edit /etc/php.ini and the below line to the same
auto_prepend_file=/var/www/codecoverage/start.php
2. codecoverage.ini
Second option is to create a 99-codecoverage.ini in /etc/php5/cli/conf.d for PHP CLI or /etc/php5/apache2/conf.d for Apache Config
I wouldn’t recommend Method #1 or #2 if you are only interested in capturing web request instead of CLI ones
3. Apache .htaccess
If you have a folder with .htaccess loading enabled, just add the below content to the file
php_value auto_prepend_file "/var/www/codecoverage/start.php"
4. Apache config
Create a /etc/apache2/conf-available/codecoverage.conf file with below content
php_value auto_prepend_file "/var/www/codecoverage/start.php"
Enable the config using the a2enconf command
$ sudo a2enconf codecoverage
$ sudo service apache2 reload
Note: This can also be applied to any specific virtual host by updating the virtual host conf file
5. Nginx Config
Nginx requests are usually forwarded to php-fpm and to pass php_value directive, we need to set a fastcgi_param
fastcgi_param PHP_VALUE "auto_prepend_file=\"/var/www/codecoverage/start.php"";
If you need to set multiple php values then look at this article for more details
Sample coverage_test.php
<?php
function test_if_else($x, $y)
{
if ($x == $y)
{
echo "X and Y are equal";
}
else
{
echo "X and Y are not equal";
}
}
test_if_else(3, 4);
Generated PHP Coverage file
When we run the coverage_test.php file through a browser, we get below coverage file generated
#coverage-unknown_test_1493209616-1493209616.4994.php
<?php
$coverage = new SebastianBergmann\CodeCoverage\CodeCoverage;
$coverage->setData(array (
'/var/www/html/test/coverage_test.php' =>
array (
5 =>
array (
0 => 'unknown_test_1493209616',
),
6 =>
array (
0 => 'unknown_test_1493209616',
),
7 =>
array (
),
8 =>
array (
),
11 =>
array (
0 => 'unknown_test_1493209616',
),
13 =>
array (
0 => 'unknown_test_1493209616',
),
15 =>
array (
0 => 'unknown_test_1493209616',
),
),
));
$coverage->setTests(array (
'unknown_test_1493209616' =>
array (
'size' => 'unknown',
'status' => NULL,
),
));
$filter = $coverage->filter();
$filter->setWhitelistedFiles(array (
'/var/www/html/test/coverage_test.php' => true,
));
Summary Report
For every request a coverage php file will get generated in /var/www/codecoverage/coverages/. Now we need a way to combine these raw PHP reports and generate a HTML format report for the same
combine.php
<?php
include_once("vendor/autoload.php");
$coverages = glob("coverages/*.php");
#increase the memory in multiples of 128M in case of memory error
ini_set('memory_limit', '12800M');
$final_coverage = new SebastianBergmann\CodeCoverage\CodeCoverage;
$count = count($coverages);
$i = 0;
foreach ($coverages as $coverage_file)
{
$i++;
echo "Processing coverage ($i/$count) from $coverage_file". PHP_EOL;
require_once($coverage_file);
$final_coverage->merge($coverage);
}
#add the directories where source code files exists
$final_coverage->filter()->addDirectoryToWhitelist("/var/www/html/");
echo "Generating final report..." . PHP_EOL;
$report = new \SebastianBergmann\CodeCoverage\Report\Html\Facade;
$report->process($final_coverage,"reports");
echo "Report generated succesfully". PHP_EOL;
?>
Running the php combine.php command in /var/www/codecoverage will generate a summary report in reports folder. The summary report would look like below
Approach 2
When I worked out Approach 1 and used with our automation suite on a large code base, it had a decent time cost. It increased request time by 2x-10x.
I was tasked to find a better approach. So I started digging into the php-code-coverage code and started debugging the whole solution to find pain points. My analysis results should 3 reasons which increased request time
- Directory whitelisting scans every file in directory
- Each PHP file is scanned to find non-exectutable lines
- During script termination the PHP report generation uses raw XDebug data and filters based on whitelist
All these steps are reptitive. What i realized is that we could afford collecting raw Debug data instead of processed code coverage data. So created a simpler start_xdebug.php
/var/www/codecoverage/start_xdebug.php
<?php
$current_dir = __DIR__;
$test_name = (isset($_COOKIE['test_name']) && !empty($_COOKIE['test_name'])) ? $_COOKIE['test_name'] : 'unknown_test_' . time();
xdebug_start_code_coverage(XDEBUG_CC_UNUSED | XDEBUG_CC_DEAD_CODE);
function end_coverage()
{
global $test_name;
global $current_dir;
$coverageName = $current_dir . '/coverages/coverage-' . $test_name . '-' . microtime(true);
try {
xdebug_stop_code_coverage(false);
$coverageName = $current_dir . '/coverages/coverage-' . $test_name . '-' . microtime(true);
$codecoverageData = json_encode(xdebug_get_code_coverage());
file_put_contents($coverageName . '.json', $codecoverageData);
} catch (Exception $ex) {
file_put_contents($coverageName . '.ex', $ex);
}
}
class coverage_dumper
{
function __destruct()
{
try {
end_coverage();
} catch (Exception $ex) {
echo str($ex);
}
}
}
$_coverage_dumper = new coverage_dumper();
Instead of generating the PHP file this time, we generate the json file. If speed is a matter, one can even use serialize function and compare the performance difference while dumping the raw data.
Since now we are dumping json data instead of php files, our summary report generation code will also change
<?php
include_once("vendor/autoload.php");
$coverages = glob("coverages/*.json");
#increase the memory in multiples of 128M in case of memory error
ini_set('memory_limit', '12800M');
$final_coverage = new SebastianBergmann\CodeCoverage\CodeCoverage;
$count = count($coverages);
$i = 0;
$final_coverage->filter()->addDirectoryToWhitelist("/var/www/html/");
foreach ($coverages as $coverage_file)
{
$i++;
echo "Processing coverage ($i/$count) from $coverage_file". PHP_EOL;
$codecoverageData = json_decode(file_get_contents($coverage_file), JSON_OBJECT_AS_ARRAY);
$test_name = str_ireplace(basename($coverage_file,".json"),"coverage-", "");
$final_coverage->append($codecoverageData, $test_name);
}
echo "Generating final report..." . PHP_EOL;
$report = new \SebastianBergmann\CodeCoverage\Report\Html\Facade;
$report->process($final_coverage,"reports");
echo "Report generated succesfully". PHP_EOL;
?>
A import change here is moving the addDirectoryToWhitelist call before loading the data, else all coverages would be filtered out
From our observation Approach 2 is 1x-1.3x.
You can download the code for above from tarunlalwani/php-code-coverage-web