A few months ago I had proposed that the Easy Digital Downloads project expand our PHP/WordPress version matrix for our Travis-CI builds. The problem we quickly ran into, is that with the existing matrix, our builds were taking ~1.25 hours to complete. This was just not a realistic means of getting feedback on a git push. In order be effective at providing feedback, it needs to be something you can wait for without using up a drastic amount of time.

Coverage For the Win/Loss

After some investigation, it was found that the biggest hit to the build performance, was code coverage. For those that don’t know, code coverage is a means of monitoring your tests, and telling you what percentage of your code is (or isn’t) tested. This allows you to build more tests to test a greater percentage of your code. This is actually a very important statistic, as it lends to the credence of your unit test passing. For example, let’s say we have two projects, Project X and Project Y. They have the following unit test pass %:

  • Project X: 100% – 10/10 Tests Passed
  • Project Y: 100% – 10/10 Tests Passed

On the surface, they both pass 10 out of 10 tests…but what if we throw code coverage into the mix:

  • Project X: 100% – 10/10 Tests Passed – 85% Code Coverage
  • Project Y: 100% – 10/10 Tests Passed – 45% Code Coverage

Are you more prone trust a codebase that has 85% of it’s code tested, and passing, or less than half? I’ll admit, we’re working to get the EDD codebase up to a better percentage of code coverage, but really that’s another story and another blog post. For now we’ll just focus on why it’s important to get code coverage.

But every time?
This is the decision we made. With Code Coverage, a single unit test took ~5 minutes to complete. Cross that with 3 PHP versions and 6 Combinations of WordPress (3 versions, with and without Multisite), and you’ve got 18 tests, @ 5 minutes each. That is a LONG time to wait for a built to complete.

Without code coverage reporting, our suite builds in ~30 seconds (on average). Now, multiply by 18 combinations of builds, and we’ve got a 9 minute build time. That’s awesome, the problem is, we were only testing PHP 5.3, 5.4, and 5.5 alongside WordPress 4.0, 4.1, and the latest nightly build. Why isn’t this enough? Well, PHP 5.6 is on the edge of being provided by some hosts, so we should get ahead of that, and our ‘Required Version’ of WordPress is 3.9.2, and we weren’t testing for it.

So how do we maximize the matrix, while maintaining code coverage reporting?

Only running Code Coverage on Pull Requests

Yep, after looking over the documentation of Travis-CI, I found a way that you could identify when a build being run is the result of a Pull Request creation/update, versus a commit to a branch. Here’s how we did it.

The Files

travis.yml
The Travis-CI config file has an entry for script (what to run as the tests) and after_script (what to run when complete). Typically you would put the BASH command you want to run here, most likely $> phpunit. Did you know though, that you can just call a normal bash script too? That’s what we did, like this:

language: php

sudo: false

php:
    - 5.3
    - 5.4
    - 5.5
    - 5.6
    - hhvm

env:
    - WP_VERSION=latest WP_MULTISITE=0
    - WP_VERSION=latest WP_MULTISITE=1
    - WP_VERSION=4.1 WP_MULTISITE=0
    - WP_VERSION=4.1 WP_MULTISITE=1
    - WP_VERSION=4.0 WP_MULTISITE=0
    - WP_VERSION=4.0 WP_MULTISITE=1
    - WP_VERSION=3.9.2 WP_MULTISITE=0
    - WP_VERSION=3.9.2 WP_MULTISITE=1

before_script:
    - bash bin/install-wp-tests.sh wordpress_test root '' localhost $WP_VERSION

script:
    - ./travis-tests.sh

after_script:
    - ./travis-after.sh

The contents of our scripts are as follows:
travis-tests.sh

#!/bin/bash
if [ "${TRAVIS_PULL_REQUEST}" = "false" ]
then
  phpunit
else
  phpunit --coverage-clover=coverage.clover
fi

travis-after.sh

#!/bin/bash
if [ "${TRAVIS_PULL_REQUEST}" != "false" ]
then
	wget https://scrutinizer-ci.com/ocular.phar
	php ocular.phar code-coverage:upload --format=php-clover coverage.clover
fi

Conclusion

What this essentially does now, is when a signal comes into Travis-CI to do a build, it checks the environment variable ${TRAVIS_PULL_REQUEST}, which when there is a pull request present is set to the pull request number, otherwise it’s false. This allows us to only run Code Coverage during a Pull Request, and for all other commits/push builds, simply run the basic unit tests, looking for breaking points.

With these faster tests, we can now have a matrix that is 5 x 8, or 40 builds, instead of 18, and finish in a fraction of the time. This gives us maximum compatibility checks, but still allows us to get coverage reports before merging in a Pull Request. Granted, our Pull Request builds take ~2 hours to complete, however these are usually items that are less necessary to be a quick response, as Pull Requests typically aren’t merged until a few hours after submitting them (at best). And on top of that, when a Pull Request is submitted, 2 builds are done, 1 Full build with code coverage, and 1 without, so we can get a quick check of it to make sure that the tests pass, and then wait to make sure we’ve got ample code coverage in the code to be merged.

So that’s just a little bit of what we decided and executed as a team to maximize our matrix, without paying the penalty of the build times. Hope you find it useful, and if you have any suggestions, I’d be happy to see them in the comments.

Cheers!


Post Image by Jean-Etienne Minh-Duy Poirrier via Creative Commons and Flickr

UPDATE: As a discussion opened on Twitter, it was also recognized by Brad Touesnard identified, we could also look into only running code coverage on one of the build matrix items. I’m going to be looking into seeing if we can optimize this way next, after I discuss it with the team.

Posted by Chris Klosowski

Chris Klosowski is the Director of Technology at Sandhills Development, LLC, creators of fine WordPress plugins, and ecommerce solutions.

36 Comments

  1. You can also exclude variations from your build matrix, to prevent running near duplicate builds. See this pull request for more detail.

    Reply

  2. Hey, I think your blog might be having browser compatibility issues.
    When I look at your blog in Ie, it looks fine but when opening
    in Internet Explorer, it has some overlapping. I just wanted to give you a quick heads up!
    Other then that, excellent blog!

    Reply

  3. I think this is a real great article.Much thanks again. Great.

    Reply

  4. Today, I went to the beachfront with my children. I found a
    sea shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She put the shell to her ear
    and screamed. There was a hermit crab inside and it pinched her ear.

    She never wants to go back! LoL I know this is totally off topic but I had to tell someone!

    Reply

  5. Heya! I’m at work surfing around your blog from my new iphone 3gs!

    Just wanted to say I love reading your blog and look forward to all your
    posts! Carry on the great work!

    Reply

  6. Amazing! Its truly remarkable piece of writing, I have got much clear idea concerning from this paragraph.

    Reply

  7. Thank you for the good writeup. It in fact
    was a amusement account it. Look advanced to far added agreeable from you!
    However, how can we communicate?

    Reply

  8. Have you ever thought about including a little bit more than just
    your articles? I mean, what you say is valuable and everything.
    However think of if you added some great photos or video clips to give your posts more, “pop”!
    Your content is excellent but with pics and clips, this blog could certainly be one of the greatest in its niche.
    Excellent blog!

    Reply

  9. Excellent blog here! Also your website a lot up very fast!
    What web host are you the use of? Can I get your associate hyperlink for your
    host? I desire my website loaded up as fast as yours lol

    Reply

  10. Useful info. Lucky me I discovered your site by chance, and I am stunned why this accident did
    not came about in advance! I bookmarked it.

    Reply

  11. Link exchange is nothing else except it is only placing the other person’s website link
    on your page at suitable place and other person will also do same
    in support of you.

    Reply

  12. I quite like looking through an article that can make men and women think.
    Also, many thanks for allowing me to comment!

    Reply

  13. Whats up are using WordPress for your blog platform?
    I’m new to the blog world but I’m trying to get
    started and create my own. Do you require any coding expertise to make your own blog?

    Any help would be greatly appreciated!

    Reply

  14. chloroquine primaquine chloroquine phosphate is hydroxychloroquine safe to take

    Reply

  15. I think the admin of this web site is really working hard for his site, because here every material
    is quality based stuff. asmr https://app.gumroad.com/asmr2021/p/best-asmr-online asmr

    Reply

  16. With havin so much content do you ever run into any issues of plagorism or copyright infringement?
    My blog has a lot of exclusive content I’ve either authored myself
    or outsourced but it looks like a lot of it is
    popping it up all over the web without my agreement. Do you know any ways to help prevent content from being stolen? I’d definitely appreciate it.
    quest bars http://j.mp/3jZgEA2 quest bars

    Reply

  17. Everything is very open with a really clear clarification of the challenges.
    It was definitely informative. Your website is very useful.
    Thank you for sharing! scoliosis surgery https://0401mm.tumblr.com/ scoliosis surgery

    Reply

  18. If some one needs expert view about running a blog afterward i suggest him/her to pay a quick visit this website, Keep up the pleasant job.
    cheap flights http://1704milesapart.tumblr.com/ cheap flights

    Reply

  19. Write more, thats all I have to say. Literally, it seems as though you relied on the video
    to make your point. You definitely know what youre talking about, why waste your intelligence
    on just posting videos to your site when you could be giving us something
    enlightening to read? ps4 games https://j.mp/3nkdKIi ps4 games

    Reply

  20. Hello to all, how is everything, I think every one is getting more from this web site,
    and your views are good designed for new viewers. quest bars https://www.iherb.com/search?kw=quest%20bars quest
    bars

    Reply

  21. Good post. I learn something totally new and challenging on blogs I stumbleupon every day.
    It will always be exciting to read through content from other authors and use something from other websites.
    scoliosis surgery https://coub.com/stories/962966-scoliosis-surgery scoliosis surgery

    Reply

Leave a reply

Your email address will not be published. Required fields are marked *