Last summer, anticipating teaching for the first time since 2013, I started reading about ungrading, and somewhere (maybe even in that piece, I didn’t check) read about a computer scientist who uses continuous integration (CI) to automatically give students feedback on their CS lab assignments. Each lab assignment has well-specified goals, and the CI automated tests evaluate the students’ solutions for correctness. Successful completion of the assignment can be counted automatically, specifications-grading style, or the instructor can review the code after it’s working for things like coding style and efficiency.

This fall I’ll be teaching a graduate methods course on data science. This seems like a great course for implementing this CI approach. But I haven’t used CI before, and the tutorials for using Travis-CI with R turned out to be unnecessarily complicated, not least because Travis-CI now has good support for R. The purpose of this post is to briefly review how to set up Github and Travis-CI for automated lab feedback.

Each lab assignment is based on this template repo: https://github.com/data-science-methods/lab-test.

Basic lab workflow

This setup assumes a workflow for lab assignments where students clone a Github repository with instructions, data, etc.; complete the assignment in a single R script; that a working solution has deterministic values for variables with set names; and that students submit their work using a pull request. A different programming language (or multiple programming languages) will require different infrastructure for running unit tests. Multiple R scripts (including more complex project structures) will require more articulated test files. Writing unit tests for non-deterministic values will be quite a bit more complicated than the example test in the template.

There are three phases for this approach: one-time setup with accounts, and then steps you and your students will do for each lab assignment.

Instructor: Account setup

You’ll only need to do these steps once.

You’ll need a Github account.
- I assume you already have one of these and that you know basic git terminology and Github workflows.
- Optional: I went ahead and created a new organization for my course, so that the course website and lab repos all live together. But that’s strictly unnecessary.
You’ll need a Travis-CI account, which you create using a Github login. Travis-CI is free for working with public Github repositories.
Optional: If you created a new organization for your course, make sure it shows up in your Travis-CI settings: Click on your profile picture (upper-right corner), then Settings. Look for the list of organizations on the left-hand column. If your organization isn’t there, then at the bottom of that column you should see a link to “Review and add your authorized organizations.”

I think that’s basically it. Travis-CI should now be able to see your public repositories.

Instructor: Repository setup

You’ll do these steps when you create each lab assignment.

In the template repo, click the green “Use this template” button (where the Clone button usually lives) to create a new repo for the lab.
- I’m going to use the naming scheme lab-01 where 01 indicates the week of the course.
Clone the new repo to the machine where you work.
Edit lab.R with the assignment instructions.
If you add any packages to the setup, add them to DESCRIPTION as well.
- Travis-CI assumes that R repositories are packages. It will automatically install dependencies, but only if all of the dependencies (including testthat) are listed in the DESCRIPTION file. You don’t need any of the usual package metadata; all you need are the list of Imports.
For each problem in the assignment, write appropriate tests in tests/test_lab.R.
- You write tests using testthat expectations: https://testthat.r-lib.org/reference/index.html.
If you changed any file names, make sure they’re consistent across lab.R, tests/test_lab.R (which source()es the assignment script), and .travis.yml (it needs to know where to point testthat::test_dir()).
Optional: Create a solutions branch. Fill in solutions for each problem, and run testthat::test_dir('tests') to check that your instructions and tests work as expected. Cherry-pick any corrections back to master.
Push master back up to Github.
In Travis-CI’s Settings, find the lab repo and flip the switch to turn on CI. (In my experience, it can take like 10 seconds for Travis-CI to see the new repo or a push/pull request to an active repo.)
Optional: Push solutions up to ensure that Travis-CI is working as expected.
- Note that there doesn’t appear to be a way to have a private branch of a public repo.

Student: Lab workflow

The students will do these steps when they work on the lab assignment.

Fork the lab assignment to their own account, then clone the fork to their working machine.
Modify the yaml header for the lab assignment with their name and so on.
Work in lab.R to complete the assignment, per instructions.
If any packages are added to the setup, add them to DESCRIPTION as well.
At any point, run testthat::test_dir('tests') to get immediate feedback on their progress.
At any point, submit a pull request to get automated feedback via Travis-CI.
- Travis-CI docs on building pull requests
Submit their work by submitting a final (passing) pull request.
Optional: Use the RStudio knitr button (or rmarkdown::render('lab.R')) to generate a pretty HTML or PDF version of their completed assignment.