Testing failure modes using error injection
Testing failure modes using error injection
When writing unit tests, it’s a good idea to test not only the happy path, but all of the failure modes as well. One way to achieve this is through error injection, also known as “failure points”.
A contrived example
Suppose we have a function that takes a string as input, splits it into chunks, and then uploads each chunk to an endpoint somewhere.
It might look something like this:
Note: Typically you would use an async function for a web request, but we’ll keep it simple for this example.
#[derive(Debug, thiserror::Error)]
pub enum UploadError {
#[error("data is invalid")]
InvalidData,
#[error("{0}")]
UploadFailed(String),
}
pub trait Uploader {
fn upload(&self, data: String) -> Result<(), UploadError>;
}
fn upload_string(input: &str, uploader: &dyn Uploader) -> Result<(), String> {
// Split the string into chunks of 3 characters each.
let chunks = input.chars().chunks(3);
// Upload each chunk.
for chunk in &chunks {
let chunk = chunk.collect::<String>();
// Try uploading each chunk a maximum of 3 times.
// If it still fails, return an error.
let mut tries_remaining = 3;
loop {
match uploader.upload(chunk.clone()) {
Ok(()) => break,
Err(e @ UploadError::UploadFailed(_)) => {
warn!("Upload failed: {e}");
tries_remaining -= 1;
if tries_remaining == 0 {
return Err(e.to_string());
}
}
Err(e) => {
return Err(e.to_string());
}
}
}
}
Ok(())
}We created an Uploader trait so that we can mock the actual upload
for testing purposes. That way we can test the logic of this function without
requiring an actual web request.
We can easily test the happy path, using a mock uploader like this:
#[test]
fn test_upload() {
let input = "This is a test";
let uploader = MockUploader::default();
upload_string(input, &uploader).unwrap();
assert_eq!(uploader.value(), input);
}We’ll get to the mock code soon, but now let’s talk about the error paths.
How do we test the following scenarios?
- One chunk fails, but then succeeds
- Multiple chunks fail, but then succeed
- A chunk fails 3 times
Error injection
We could just add a flag in the mock for each scenario, and then use if
conditions everywhere to make it work. But this moves our test logic outside
the tests and it’s not very reusable.
What if there was a way to describe the desired failure scenario in the test, and then verify the results of that scenario? Even better, what if it could be generic over the error type so that it could be easily reused?
First, let’s define an ErrorTrigger. It’s generic over the error type so
that it can be easily reused. Its job is to track how many times it has been
invoked and conditionally construct and return an error if required.
We also need an ErrorFrequency so we know when to trigger the error.
pub enum ErrorFrequency {
Never,
Always,
Sequence(HashSet<usize>),
}
impl ErrorFrequency {
pub fn should_trigger(&self, count: usize) -> bool {
match self {
Self::Never => false,
Self::Always => true,
Self::Sequence(x) => x.contains(&count),
}
}
}
pub struct ErrorTrigger<E> {
/// The number of times the trigger has been called.
count: usize,
/// The number of times the trigger returned an error.
trigger_count: usize,
/// When to return an error.
frequency: ErrorFrequency,
/// Closure that creates the error.
error_fn: Box<dyn Fn() -> E>,
}
impl<E> ErrorTrigger<E> {
pub fn new<F>(frequency: ErrorFrequency, error_fn: F) -> Self
where
F: Fn() -> E + 'static,
{
Self {
count: 0,
trigger_count: 0,
frequency,
error_fn: Box::new(error_fn),
}
}
pub fn tick(&mut self) -> Result<(), E> {
self.count += 1;
if self.frequency.should_trigger(self.count) {
self.trigger_count += 1;
return Err((self.error_fn)());
}
Ok(())
}
}Now, we can construct the trigger in our test by specifying exactly when it
should fail, and call the tick() method from our mock uploader and it will
“inject” the error according to the schedule we predefined in the test.
Before we create the test, let’s take a look at the code for the mock uploader.
Creating the mock
Remember the Uploader trait from our first code snippet? Let’s create a mock
that implements that trait.
The mock collects the chunks that are uploaded and then stores them in a vec.
// Using mutexes so that we can modify the values behind &self.
#[derive(Default)]
pub struct MockUploader {
received: Mutex<Vec<String>>,
error_trigger: Mutex<Option<ErrorTrigger<UploadError>>>,
}
impl Uploader for MockUploader {
fn upload(&self, data: String) -> Result<(), UploadError> {
// Optionally return an error, if specified by the test.
if let Some(trigger) = self.error_trigger.lock().expect("lock poisoned").as_mut() {
trigger.tick()?;
}
// Save each chunk to verify later.
self.received.lock().expect("lock poisoned").push(data.to_string());
Ok(())
}
}That’s all we need to make the error trigger work. There is some boilerplate
to lock the mutex but essentially we just call the tick() method and bubble
up the error if there was one.
Let’s add some helper functions to the mock to make our test code nicer.
impl MockUploader {
/// Add an `ErrorTrigger` to the mock.
pub fn with_error_trigger(mut self, trigger: ErrorTrigger<UploadError>) -> Self {
self.error_trigger = Mutex::new(Some(trigger));
self
}
/// Join the received chunks back into a string.
pub fn value(&self) -> String {
self.received.lock().expect("lock poisoned").join("")
}
/// Get the number of times the mock returned an error.
pub fn error_trigger_count(&self) -> usize {
self.error_trigger
.lock()
.expect("lock poisoned")
.as_ref()
.map(|x| x.trigger_count)
.unwrap_or_default()
}
}Writing the tests
Now we’re ready to write the tests.
We said we wanted to test these scenarios:
- One chunk fails, but then succeeds
- Multiple chunks fail, but then succeed
- First chunk fails 3 times
Writing tests for these scenarios is now quite straightforward. All we need to
do is construct the required error trigger and then pass the mock to our
upload_string() function.
#[test]
fn test_upload_transient_fail() {
let input = "This is a test";
let uploader = MockUploader::default().with_error_trigger(ErrorTrigger::new(
ErrorFrequency::Sequence(HashSet::from([1])),
|| UploadError::UploadFailed("network error".to_string()),
));
upload_string(input, &uploader).unwrap();
assert_eq!(uploader.value(), input);
assert_eq!(uploader.error_trigger_count(), 1);
}
#[test]
fn test_upload_multiple_transient_fail() {
let input = "This is a test";
let uploader = MockUploader::default().with_error_trigger(ErrorTrigger::new(
ErrorFrequency::Sequence(HashSet::from([1, 4, 7])),
|| UploadError::UploadFailed("network error".to_string()),
));
upload_string(input, &uploader).unwrap();
assert_eq!(uploader.value(), input);
assert_eq!(uploader.error_trigger_count(), 3);
}
#[test]
fn test_upload_first_chunk_fail() {
let input = "This is a test";
let uploader = MockUploader::default()
.with_error_trigger(ErrorTrigger::new(ErrorFrequency::Always, || {
UploadError::UploadFailed("network error".to_string())
}));
let e = upload_string(input, &uploader).unwrap_err();
assert_eq!(e, "network error");
assert_eq!(uploader.error_trigger_count(), 3);
}Wrapping up
This example shows how you can inject errors into a mock in a way that is highly reusable. It puts the test condition and logic in the test itself, making each test more readable.
There are many ways to extend this approach by providing more helper methods on
the ErrorTrigger struct, and even creating a wider-scoped ErrorPlan struct
that can be shared across your whole application, for more complex error
scenarios. You also might want to do other things besides return an error,
such as simulate delays or other failure modes.
Error injection is a very powerful tool for testing your error paths and improving your test coverage.
Thanks for reading.