%\VignetteIndexEntry{Using tinytest} \documentclass[11pt]{article} \usepackage{enumitem} \usepackage{xcolor} % for color definitions \usepackage{sectsty} % to modify heading colors \usepackage{fancyhdr} \setlist{nosep} % simpler, but issue with your margin notes \usepackage[left=1cm,right=3cm, bottom=2cm, top=1cm]{geometry} %\usepackage{vmargin} %\setpapersize{USletter} % or a4 for you % Left Top Right Bottom headheight?? headsep footheight footsep %\setmarginsrb{0.75in}{0.25in}{1.1in}{0.25in}{15pt}{0pt}{10pt}{20pt} \usepackage{hyperref} \definecolor{bluetext}{RGB}{0,101,165} \definecolor{graytext}{RGB}{80,80,80} \hypersetup{ pdfborder={0 0 0} , colorlinks=true , urlcolor=blue , linkcolor=bluetext , linktoc=all , citecolor=blue } \sectionfont{\color{bluetext}} \subsectionfont{\color{bluetext}} \subsubsectionfont{\color{bluetext}} % no serif=better reading from screen. \renewcommand{\familydefault}{\sfdefault} % header and footers \pagestyle{fancy} \fancyhf{} % empty header and footer \renewcommand{\headrulewidth}{0pt} % remove line on top \rfoot{\color{bluetext} tinytest \Sexpr{packageVersion("tinytest")}} \lfoot{\color{black}\thepage} % side-effect of \color{}: lowers the printed text a little(?) \usepackage{fancyvrb} % custom commands make life easier. \newcommand{\code}[1]{\texttt{#1}} \newcommand{\pkg}[1]{\textbf{#1}} \let\oldmarginpar\marginpar \renewcommand{\marginpar}[1]{\oldmarginpar{\color{bluetext}\raggedleft\scriptsize #1}} % skip line at start of new paragraph \setlength{\parindent}{0pt} \setlength{\parskip}{1ex} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \title{Using tinytest} \author{Mark van der Loo} \date{\today{} | Package version \Sexpr{packageVersion("tinytest")}} \begin{document} \DefineVerbatimEnvironment{Sinput}{Verbatim}{fontshape=n,formatcom=\color{graytext}} \DefineVerbatimEnvironment{Soutput}{Verbatim}{fontshape=sl,formatcom=\color{graytext}} \newlength{\fancyvrbtopsep} \newlength{\fancyvrbpartopsep} \makeatletter \FV@AddToHook{\FV@ListParameterHook}{\topsep=\fancyvrbtopsep\partopsep=\fancyvrbpartopsep} \makeatother \setlength{\fancyvrbtopsep}{0pt} \setlength{\fancyvrbpartopsep}{0pt} \maketitle{} \thispagestyle{empty} \tableofcontents{} <>= options(prompt="R> ", continue = " ", width=75, tt.pr.color=FALSE) @ \subsection*{Reading guide} Readers of this document are expected to know how to write R functions and have a basic understanding of a package source directory structure. \newpage{} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Purpose of this package: unit testing} The purpose of \emph{unit testing} is to check whether a function gives the output you expect, when it is provided with certain input. So unit testing is all about comparing \emph{desired} outputs with \emph{realized} outputs. The purpose of this package is to facilitate writing, executing and analyzing unit tests. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Expressing tests} Suppose we define a function translating pounds (lbs) to kilograms inaccurately. <<>>= lbs2kg <- function(x){ if ( x < 0 ){ stop(sprintf("Expected nonnegative weight, got %g",x)) } x/2.20 } @ We like to check a few things before we trust it. <<>>= library(tinytest) expect_equal(lbs2kg(1), 1/2.2046) expect_error(lbs2kg(-3)) @ The value of an \code{expect\_*} function is a \code{logical}, with some attributes that record differences, if there are any. These attributes are used to pretty-print the results. <<>>= isTRUE( expect_true(2 == 1 + 1) ) @ \subsection{Test functions} Currently, the following expectations are implemented. \begin{center} \begin{tabular}{ll} \textbf{Function} & \textbf{what it expects}\\ \code{expect\_equal(current, target)} & equality (using \code{all.equal})\\ \code{expect\_equivalent(current, target)} & equality, ignoring attributes\\ \code{expect\_identical(current, target)} & equality, (using, \code{identical})\\ \code{expect\_length(current, length)} & check lenght of object\\ \code{expect\_true(current)} & \code{current} evaluates to \code{TRUE}\\ \code{expect\_false(current)} & \code{current} evaluates to \code{FALSE}\\ \code{expect\_match(current, pattern)} & All strings in \code{current} match \code{pattern}\\ \code{expect\_inherits(current, class)} & \code{current} inherits from \code{class}\\ \code{expect\_null(current)} & \code{current} evaluates to \code{NULL}\\ \code{expect\_error(current, pattern)} & error message matching \code{pattern}\\ \code{expect\_warning(current, pattern)} & warning message matching \code{pattern}\\ \code{expect\_message(current, pattern)} & message matching \code{pattern}\\ \code{expect\_silent(current)} & expect no warnings or errors (just run)\\ \code{expect\_stdout(current, pattern)} & expect output to \code{stdout} matching \code{pattern}\\ \code{expect\_equal\_to\_reference(current, file)} & expect object equal to object stored in RDS file\\ \code{expect\_equivalent\_to\_reference(current, file)} & expect object equivalent to object stored in RDS file\\ \end{tabular} \end{center} Here, \code{target} is the intended outcome and \code{current} is the observed outcome. Also, \code{pattern} is interpreted as a regular expression. <<>>= expect_error(lbs2kg(-3), pattern="nonnegative") expect_error(lbs2kg(-3), pattern="foo") @ \subsection{Alternative syntax} The syntax of the test functions should be familiar to users of the \code{testthat} package\cite{wickham2016testthat}. In test files only, you can use equivalent functions in the style of \code{RUnit}\cite{burger2016RUnit}. To be precise, for each function of the form \code{expect\_lol} there is a function of the form \code{checkLol}. \subsection{Interpreting the output and print options} Let's have a look at an example again. <<>>= expect_false( 1 + 1 == 2, info="My personal message to the tester" ) @ The output of these functions is pretty self-explanatory, nevertheless we see that the output of these expect-functions consist of \begin{itemize} \item The result: \code{FAILED}, \code{PASSED} or \code{SIDEFX}. The latter only occurs when side effects are monitored (see \S\ref{sect:side}) \item The type of failure (if any) between square brackets. Current options are as follows. \begin{itemize} \item \code{[data]} there are differences between observed and expected values. \item \code{[attr]} there are differences between observed and expected attributes, such as column names. \item \code{[xcpt]} an exception (warning, error) was expected but not observed. \end{itemize} When side effects are monitored, and the result is \code{SIDEFX}, a side effect was observed. The type of side effect is reported between square brackets. \begin{itemize} \item \code{[envv]} An environmental variable was created, changed, or deleted. \item \code{[wdir]} The working directory has changed. \item \code{[file]} A file operation occurred in the test directory or one of its subdirectories. \end{itemize} \item When relevant (see \S\ref{sect:testfiles}), the location of the test file and the relevant line numbers. \item The test call. \item When relevant, a summary of the differences between observed and expected values or attributes, or a summary of the observed side effect. \item When present, a user-defined information message. \end{itemize} The result of an \code{expect\_} function is a \code{tinytest} object. You can print them in long format (default) or in short, one-line format like so. <<>>= print(expect_equal(1+1, 3), type="short") @ \marginpar{\code{print} method} Functions that run multiple tests return an object of class \code{tinytests} (notice the plural). Since there may be a lot of test results, \pkg{tinytest} tries to be smart about printing them. The user has ultimate control over this behaviour. See \code{?print.tinytests} for a full specification of the options. \section{Test files} \label{sect:testfiles} In \pkg{tinytest}, tests are scripts, interspersed with statements that perform checks. An example test file in tinytest can look like this. \begin{verbatim} # contents of test_addOne.R addOne <- function(x) x + 2 expect_true(addOne(0) > 0) hihi <- 1 expect_equal(addOne(hihi), 2) \end{verbatim} A particular file can be run using\marginpar{\code{run\_test\_file}} <>= run_test_file("test_addOne.R", verbose=0) @ We use \code{verbose=0} to avoid cluttering the output in this vignette. By default, verbosity is turned on, and a counter is shown while tests are run. The counter is colorized on terminals supporting ANSI color extensions. If you are uncomfortable reading these colors or prefer colorless output, use \code{color=FALSE} or set \code{options(tt.pr.color=FALSE)}. The numbers between \code{<-->} indicate at what lines in the file the failing test can be found. By default only failing tests are printed. You can store the output and print all of them. <<>>= test_results <- run_test_file("test_addOne.R", verbose=0) print(test_results, passes=TRUE) @ Or you can set <>= options(tt.pr.passes=TRUE) @ to print all results during the active R session. To run all test files in a certain directory, we can use\marginpar{\code{run\_test\_dir}} <>= run_test_dir("/path/to/your/test/directory") @ By default, this will run all files of which the name starts with \code{test\_}, but this is customizable. \subsection{Summarizing test results, getting the data} To create some results, run the tests in this package. <<>>= out <- run_test_dir(system.file("tinytest", package="tinytest") , verbose=0) @ The results can be turned into data using \code{as.data.frame}. \marginpar{\code{as.data.frame}} <<>>= head(as.data.frame(out), 3) @ The last two columns indicate the line numbers where the test was defined. A `summary` of the output gives a table with passes and fails per file. \marginpar{\code{summary}} <<>>= summary(out) @ \subsection{Programming over tests, ignoring test results, exiting early} Test scripts are just R scripts interspersed with tests. The test runners make sure that all test results are caught, unless you tell them not to. For example, since the result of a test is a \code{logical} you can use them as a condition. <>= if ( expect_equal(1 + 1, 2) ){ expect_true( 2 > 0) } @ Here, the second test (\code{expect\_true(2 > 0)}) is only executed if the first test results in \code{TRUE}. In any case the result of the first test will be caught in the test output, when this is run with \code{run\_test\_file} \code{run\_test\_dir}, \code{test\_all}, \code{build\_install\_test} or through \code{R CMD check} using \code{test\_package}. If you want to perform the test, but not record the test result you can do the following \marginpar{\code{ignore}} (note the placement of the brackets). <>= if ( ignore(expect_equal)(1+1, 2) ){ expect_true(2>0) } @ Other cases where this may be useful is to perform tests in a loop, e.g. when there is a systematic set of cases to test. It is possible to exit a test file prematurely. For example when there are a number of tests that are not relevant or possible on some OS, you can do the following. \marginpar{\code{exit\_file}} <<>>= if ( Sys.info()[['sysname']] == "Windows"){ exit_file("Cannot test this on Windows") } @ This will cause \code{run\_test\_file} to stop file execution, print the message, and report the information gathered up to where \code{exit} was called. A function like \code{test\_all} will then continue with the next file, so testing is not aborted completely. There is also a convenience function called \code{exit\_if\_not} that functions similarly to base R's \code{stopifnot}. It accepts a comma-separated list of expressions, and if any one of them does not result in a single \code{TRUE} the execution of the test file is stopped with a message. So you can do things like this. <>= exit_if_not(requireNamespace("slartibartfast", quietly=TRUE) , packageVersion("slartibartfast") >= "1.0.0") @ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Running order and side effects} \label{sect:running} It is a generally a good idea to write test files that are independent from each other. This means that the order of running them is unimportant for the test results and test files can be maintained independently. The function \code{run\_test\_file} and by extension \code{run\_test\_dir}, \code{test\_all}, and \code{test\_package} encourage this by resetting \begin{itemize} \item options, set with \code{options()}; \item environment variables, set with \code{Sys.setenv()} \end{itemize} after a test file is executed. To escape this behavior, use \code{base::Sys.setenv()} respectively \code{base::options()}. Alternatively use <>= run_test_dir("/path/to/my/testdir" , remove_side_effects = FALSE) test_all("/path/to/my/testdir" , remove_side_effects = FALSE) # Only in tests/tinytest.R: test_package("PACKAGENAME", remove_side_effects=FALSE) @ Test files are sorted and run based on the current locale. This means that the order of execution is in general not platform-independent. You can control the sorting behavior interactively or by setting \code{options(tt.collate)}. To be precise, adding <>= options(tt.collate="C") @ to \code{/tests/tinytest.R} before running \code{test\_package} will ensure bytewise sorting on most systems. See also \code{help("run\_test\_dir")}. \subsection{Monitoring side effects} \label{sect:side} The term 'side effect' is the technical expression for the situation where a function or expression changes something outside of its scope. Examples include creating, removing, or changing variables in R's global work space, R options, or environment variables of your operating system. We will call such variables or options \emph{external variables}. To test for side-effects once, use the \code{side\_effects} argument to any of the test runners. For example <>= test_package("pkg", side_effects=TRUE) @ There is control over which side-effects to track. For example to prevent tracking changes in the working directory, do the following. <>= test_package("pkg", side_effects=list(pwd=FALSE)) @ If you add \code{report\_side\_effects()} anywhere in a test file, certain external variables are monitored from that point on, and for that file only. It can be switched off again by calling \code{report\_side\_effects(FALSE)} anywhere in the file. The reporting functionality will compare the external state before and after every expression in the test file is run and report any changes. At the moment, effects that can be monitored include environment variables, locale settings, the present working directory, and file operations in the test directory. Below is an example of a test file where side effects are recorded. The third line creates an explicit side effect by creating a new environment variable called \code{hihi} with the value \code{"lol"}. \begin{verbatim} # contents of test_se.R report_side_effects() expect_equal(1+1, 2) Sys.setenv(hihi="lol") expect_equal(1+1, 3) Sys.setenv(hihi="lulz ftw") \end{verbatim} Running the test file yields an object of class \code{tinytests} as usual, only now changes in environment variables are reported. <<>>= run_test_file("test_se.R", verbose=1) @ Note that as discussed in Section~\S\ref{sect:running}, \pkg{tinytest} will unset the environment variable \code{hihi} automatically after running the file because it was set directly by the author of the test file using \code{Sys.setenv}. The real value of the reporting functionality is that it also reports on external variables that are touched by other functions than those you call explicitly in the file. Reading and comparing versions of external variables takes some time. Especially when it requires a call to the operating service such as a request for values of environment variables. We therefore recommend this to be used only when you suspect a side effect. Or, for example to execute \code{report\_side\_effects()} conditional on \code{at\_home()}. It is not possible to catch all types of side effects, even in principle, using the \pkg{tinytest} reporting functionality. Examples include: packages that keep a global variable or environment within their namespace to store some state, and packages that rely on compiled code where there are global objects within the shared object. Side effects are to be avoided as a general and strong principle, but sometimes there is little or no choice. In Section~\ref{sect:devil} we give some tips on how to properly handle such situations. \section{Testing packages} Using \pkg{tinytest} for your package is pretty easy. \begin{enumerate} \item Testfiles are placed in \code{/inst/tinytest}. The testfiles all have names starting with \code{test} (for example \code{test\_haha.R}). \item In the file \code{/tests/tinytest.R} you place the code \begin{verbatim} if ( requireNamespace("tinytest", quietly=TRUE) ){ tinytest::test_package("PACKAGENAME") } \end{verbatim} \item In your \code{DESCRIPTION} file, add \pkg{tinytest} to \code{Suggests:}. \end{enumerate} You can automatically create a minimal running test infrastructure with the \code{setup\_tinytest} function. \marginpar{\code{setup\_tinytest}} <>= setup_tinytest("/path/to/your/package") @ In a terminal, you can now do \begin{verbatim} R CMD build /path/to/your/package R CMD check PACKAGENAME_X.Y.Z.tar.gz \end{verbatim} and all tests will run. To run all the tests interactively, make sure that all functions of your new package are loaded. After that, run\marginpar{\code{test\_all}} <>= test_all("/path/to/your/package") @ where the default package directory is the current working directory. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Build--install--test interactively} The most realistic way to unit-test your package is to build it, install it and then run all the tests. The function <>= build_install_test("/path/to/your/package") @ does exactly that. It builds and installs the package in a temporary directory, starts a fresh R session, loads the newly installed package and runs all tests. The return value is a \code{tinytests} object. The package is built without manual or vignettes to speed up the whole process. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Testing functions that are not exported: use `:::`} In Section~\ref{sect:spherical} it is argued that unit tests should as a rule of thumb focus on the functions that are visible to the user. However, there are cases where it may be preferred to test an internal function. For example when there are two user-visible functions that call the same underlying, unexported function with different arguments. Or when one of the internal functions implements a numerical algorithm that requires thorough testing. To test functions in your package that are not visible to users that load your package, use the triple-colon operator like so. <>= output = pkg:::some_internal_function(1) expect_equal(output, 2) @ This is perfectly ok, and is also accepted by \code{R CMD check --as-cran}. \pkg{Tinytest} does not make those internal functions directly callable like some other unit testing packages do. Making internal functions callable means that \begin{enumerate} \item \pkg{tinytest} needs to simulate loading a package, except for the namespace restrictions; \item there may be significant differences between the environment in which you test the functions, and the environment which a user sees when loading the package; \item the way the package is loaded during testing may differ from the way it is loaded when a user loads it; \item exported functions are not clearly distinguished from internal functions in the test code. \end{enumerate} accurately simulating how R loads a package is no small matter. It would require a significant epansion of \code{tinytest}'s code base that would have to be kept synchronised with the way R loads packages (possibly with backport options when R would change in that area). So in short: let's keep things simple and let R do what it knows how to do. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Using data stored in files} \label{sect:comparefiles} When your package is tested with \code{test\_package}, \pkg{tinytest} ensures that your working directory is the testing directory (by default \code{tinytest}). This means you can read files that are stored in your folder directly. Suppose that your package directory structure looks like this (default): \begin{itemize} \item[]\code{/inst} \begin{itemize} \item[]\code{/tinytest} \begin{itemize} \item[]\code{/test.R} \item[]\code{/women.csv} \end{itemize} \end{itemize} \end{itemize} Then, to check whether the contents of \code{women.csv} is equal to the built-in \code{women} dataset, the content of \code{test.R} looks as follows. <>= dat <- read.csv("women.csv") expect_equal(dat, women) @ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Skipping tests on \code{CRAN}} It is not possible to detect whether a test is running on CRAN. This means we are forced to detect that we are running tests in our own environment. In the following example we use the host name to detect if we are running on our own machine and explicitly pass this information to \code{test\_package}. <>= options(prompt=" ", continue=" ") @ <>= # contents of pkgdir/tests/tinytest.R if ( requireNamespace("tinytest", quietly=TRUE) ){ home <- identical( Sys.info()["nodename"], "YOURHOSTNAME" ) tinytest::test_package("PKGNAME", at_home = home) } @ Other ways to detect whether you are running `at home' include \begin{itemize} \item Set a custom environment variable (from your OS) and detect it with \code{Sys.getenv}. <<>>= home <- identical( Sys.getenv("HONEYIMHOME"), "TRUE" ) @ \item Use 4-number package versioning while developing and 3-number versioning for CRAN releases\footnote{As \href{https://stackoverflow.com/questions/36166288/skip-tests-on-cran-but-run-locally}{recommended here} by Dirk Eddelbuettel.}. <>= home <- length(unclass(packageVersion("PKGNAME"))[[1]]) == 4 @ \end{itemize} <>= options(prompt="R> ", continue=" ") @ \subsubsection*{When tests are run interactively} All the interactive test runners have \code{at\_home=TRUE} by default, so while you are developing all tests are run, unless you exclude them explicitly. <<>>= run_test_file("test_hehe.R", verbose=0) run_test_file("test_hehe.R", verbose=0, at_home=FALSE) @ Here is an overview of test runners and their default setting for \code{at\_home}. \begin{center} \begin{tabular}{rll} Function & Default \code{at\_home} & Intended use \\ \hline \code{run\_test\_file} & \code{TRUE} & Interactive by developer\\ \code{run\_test\_dir} & \code{TRUE} & Interactive by developer\\ \code{test\_all} & \code{TRUE} & Interactive by developer\\ \code{build\_install\_test} & \code{TRUE} & Interactive by developer\\ \code{test\_package} & \code{FALSE} & \code{R CMD check}, or after installation by user. \end{tabular} \end{center} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Testing your package after installation} Supposing your package is called \pkg{hehe} and the \pkg{tinytest} infrastructure is used. If the package is installed, the following command runs \pkg{hehe}'s tests. <>= tinytest::test_package("hehe") @ This can come in handy when a user of \pkg{hehe} reports a bug and you want to make sure all tested functionality works on the user's system. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Using extension packages} It is possible for other packages to add custom assertions (\code{expect}-functions). To use such a package: \begin{enumerate} \item Add the extension package to the \code{Suggests:} field in the \code{DESCRIPTION} file. \item Add \code{using(pkg)} to \emph{each} test file that use the extensions (see \code{?using}). \marginpar{\code{using}} \end{enumerate} When multiple extension packages are loaded, and when there are name collisions, the packages loaded later takes precedence over the ones loaded earlier (as usual in R). This includes assertions exported by \pkg{tinytest}. \textbf{Note.} Other than in regular R, it is not possible to disambiguate functions using namespace resolution as in \code{pkg::expect\_something}, because in that case the test result will not be caught by \pkg{tinytest}. The API for building extension packages is described in \code{?register\_tinytest\_extension}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Mocking databases} The \pkg{dittodb} package\cite{keane2020dittodb} is capable of mocking pre-recorded database requests. To use extensions like \code{dittodb}, put the package in \code{Suggests:} in the \code{DESCRIPTION} and load it in the test file. The package captures responses from SQL connections and saves them to R files. Therefore, capture the requested response prior to testing by wrapping your request in \code{start\_db\_capturing()} and \code{stop\_db\_capturing()}. Optionally, specify the file path you want your mocks to be saved. For examples using \code{dittodb}, see \href{https://dittodb.jonkeane.com/}{here}. In the testing scripts, load the package and wrap your tests in \code{with\_mock\_path(path, with\_mock\_db())} like <>= require(dittodb) with_mock_path( system.file("", package = "myPackage"), with_mock_db({ # }) ) @ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Testing with environmental variables} \label{sect:envvar} In \pkg{tinytest} you can run tests with custom environment variable settings easily. Just add the \code{set\_env} argument to any of \code{run\_test\_file()}, \code{run\_test\_dir()} or \code{test\_package()}. Here is an example. <>= test_package("tinytest", set_env = list(WA_BABALOOBA="BA_LA_BAMBOO")) @ With this option, the environment variable will be set during testing and unset afterwards. Setting and unsetting environment variables like this will not be recorded as a side effect. If there is code in the test file that changes this variable, then it is recorded. Do note that R uses some environment variables that are read during startup, such as \code{\_R\_OPTIONS\_STRINGS\_AS\_FACTORS\_}. Setting these at runtime has no effect. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Running tests in parallel} In \pkg{tinytest}, a file should be considered a closed unit: no information created in one test file should be used in another. Under this condition, tests can automatically run in parallel by running different files in different R sessions. Running code in parallel takes some careful consideration around setting up a cluster, running the tests, and closing the cluster of preparing it for the next run. Depending on the test runner used, there are different levels of control and responsibility for the user to prepare the program for parallelization. Below we describe them from less easier to more control. \subsubsection*{\code{build\_install\_test}} This function creates and installs a package in a temporary location. By setting the \code{ncpu} parameter, the number of cores used at the testing phase can be increased. <>= build_install_test("/path/to/your/package", ncpu=2) @ We already mentioned that the order in which files are run is in principle system-dependent and it is a good practice not to rely on it. Under parallel situations, all bets on file order are off. \subsubsection*{\code{test\_package}.} This function assumes that a package is installed. It can gather any information necessary to parallelize a test run. The simplest way to parallelize is to specify the number of CPUs used. <>= test_package("PACKAGENAME", ncpu=2) @ Here, \code{test\_package} will \begin{enumerate} \item Set up a local cluster using \code{parallel::makeCluster}. \item Load the package on each R instance of the cluster. \item Run test files in parallel over the cluster. \item Collect the results and close the cluster. \end{enumerate} In stead of just passing the number of CPUs it is possible to pass a \code{cluster} object. In that case \code{test\_package} will still load the package on each node. However, note that if the package gets updated and reinstalled, it should also be reloaded. It is in general hard to completely unload a package in R (see \code{?detach} and \code{?unloadNamespace} for some details on artifacts that will not be removed). So our advice is to restart a cluster for each test run. \subsubsection*{\code{run\_test\_dir}, \code{test\_all}} These function assumes that all functionality needed to run the tests is loaded. They accept an object of type \code{cluster}. The user is responsible for setting up the nodes. <>= cl <- parallel::makeCluster(4, outfile="") parallel::clusterCall(cl, source, "R/myfunctions.R") run_test_dir("inst/tinytest", cluster=cl) @ where the argument \code{outfile=""} ensures that messages from each node are forwarded to the master node. It is possible to keep the cluster `alive', so modifications can be made to \code{"R/myfunctions.R"} and then run for example the following. <>= parallel::clusterCall(cl, source, "R/myfunctions.R") test_all(cluster=cl) stopCluster(cl) @ For heavy test routines it is thus possible to keep a test cluster up to offload computations. For more complex situations, including packages that use \code{S4} classes, or compiled code, (re)loading takes more effort than sourcing a few R files. In this cases it is often easier to restart a clean cluster for each test round. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{A few tips on packages and unit testing} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Make your package spherical} \label{sect:spherical} Larger packages typically consist of functions that are visible to the users (exported functions) and a number of functions that are only used by the exported functions. For example: <<>>= # exported, user-visible function inch2cm <- function(x){ x*conversion_factor("inch") } # not exported function, package-internal conversion_factor <- function(unit){ confac <- c(inch=2.54, pound=1/2.2056) confac[unit] } @ We can think of the exported functions as the \emph{surface} of the package and all the other functions as the \emph{volume}. The surface is what a user sees, the volume is what the developer sees. The surface is how a user interacts with a package. If the surface is small (few functions exported), users are limited in the ways they can interact with your package and that means there is less to test. So as a rule of thumb, it is a good idea to keep the surface small. Since a sphere has the smallest surface-to-volume ratio possible, I refer to this rule as \emph{keep your package spherical}. By the way, the technical term for the surface of a package is API (application program interface). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Test the surface, not the volume} Unexpected behavior (a bug) is often discovered when someone who is not the developer starts using code. Bugfixing implies altering code and it may even require you to refactor large chunks of code that is internal to a package. If you defined extensive tests on non-exported functions, this means you need to rewrite the tests as well. As a rule of thumb, it is a good idea to test only the behaviour at the surface, so as a developer you have more freedom to change the internals. This includes rewriting and renaming internal functions completely. By the way, it is bad practice to change the surface, since that means you are going to break other people's code. Nobody likes to program against an API that changes frequently, and everybody hates to program against an API that changes unexpectedly. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{How many tests do I need?} When you call a function, you can think of its arguments flowing through a certain path from input to output. As an example, let's take a look again at a new, slightly safer unit conversion function. <<>>= pound2kg <- function(x){ stopifnot( is.numeric(x) ) if ( any(x < 0) ){ warning("Found negative input, converting to positive") x <- abs(x) } x/2.2046 } @ If we call \code{lbs2kg} with argument \code{2}, we can write: \begin{verbatim} 2 -> /2.2046 -> output \end{verbatim} If we call \code{lbs2kg} with argument \code{-3} we can write \begin{verbatim} -3 -> abs() -> /2.2046 -> output \end{verbatim} Finally, if we call \code{pound2kg} with \code{"foo"} we can write \begin{verbatim} "foo" -> stop() -> Exception \end{verbatim} So we have three possible paths. In fact, we see that every nonnegative number will follow the first path, every negative number will follow the second path and anything nonnumeric follows the third path. So the following test suite fully tests the behaviour of our function. <>= expect_equal(pound2kg(1), 1/2.2046 ) # test for expected warning, store output expect_warning( out <- pound2kg(-1) ) # test the output expect_equal( out, 1/2.2046) expect_error(pound2kg("foo")) @ The number of paths of a function is called its \emph{cyclomatic complexity}. For larger functions, with multiple arguments, the number of paths typically grows extremely fast, and it quickly becomes impossible to define a test for each and every one of them. If you want to get an impression of how many tests one of your functions in needs in principle, you can have a look at the \pkg{cyclocomp} package of G\'abor Cs\'ardi\cite{csardi2016cyclocomp}. Since full path coverage is out of range in most cases, developers often strive for something simpler, namely \emph{full code coverage}. This simply means that each line of code is run in at least one test. Full code coverage is no guarantee for bugfree code. Besides code coverage it is therefore a good idea to think about the various ways a user might use your code and include tests for that. To measure code coverage, I recommend using the \pkg{covr} package by Jim Hester\cite{hester2018covr}. Since \pkg{covr} is independent of the tools or packages used for testing, it also works fine with \pkg{tinytest}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{It's not a bug, it's a test!} If users of your code are friendly enough to submit a bug report when they find one, it is a good idea to start by writing a small test that reproduces the error and add that to your test suite. That way, whenever you work on your code, you can be sure to be alarmed when a bug reappears. Tests that represent earlier bugs are sometimes called \emph{regression tests}. If a bug reappears during development, software engineers sometimes refer to this as a \emph{regression}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Side effects are the Devil} \label{sect:devil} Since side-effects manipulate variables outside of the scope of a function, or even outside of R, they can cause bugs that are hard to reproduce. R offers a mechanism to ensure that a function leaves the outside world as it was, once your code stops running. Suppose you need to change working directory within a function, \code{source} a file and return to the working directory. A naive way to do this is like so. \marginpar{\code{on.exit}} <<>>= bad_function <- function(file){ oldwd <- getwd() setwd(dirname(file)) source(basename(file)) setwd(oldwd) } @ This all works fine, untill \code{file} contains faulty code and throws an error. As result, the execution of \code{bad\_function} will stop and leave the user in a changed working directory. With \code{on.exit} you can define code that will be carried out before the function exits, either normally or with an error. <<>>= good_function <- function(file){ oldwd <- getwd() on.exit(setwd(oldwd)) setwd(dirname(file)) source(basename(file)) } @ \newpage \begin{thebibliography}{5} \bibitem{wickham2016testthat} \href{https://cran.r-project.org/package=testthat}{Unit Testing for R} Hadley Wickham (2016). testthat: Get Started with Testing. The R Journal, vol. 3, no. 1, pp. 5--10, 2011 \bibitem{burger2016RUnit} Matthias Burger, Klaus Juenemann and Thomas Koenig (2018). \href{https://CRAN.R-project.org/package=RUnit}{RUnit: R Unit Test Framework} R package version 0.4.32. \bibitem{csardi2016cyclocomp} \href{https://cran.r-project.org/package=cyclocomp}{cyclocomp: cyclomatic complexity of R code} G\'abor Cs\'ardi (2016). R package version 1.1.0 \bibitem{hester2018covr} \href{https://CRAN.R-project.org/package=covr}{covr: Test Coverage for Packages} Jim Hester (2018). R package version 3.2.1 \bibitem{keane2020dittodb} \href{https://CRAN.R-project.org/package=dittodb}{dittodb: A Test Environment for Database Requests} Jonathan Keane and Mauricio Vargas (2020). R package version 0.1.1 \end{thebibliography} \end{document}