Using OpenCV and NdArray in Rust

Jul 28, 2025

Introduction

This article describes some techniques for mixing the ndarray and opencv crates in Rust when doing image processing.

If you do image processing in Python, you have almost certainly used the cv2 and numpy Python libraries.

When switching to Rust, the equivalent crates are opencv and ndarray. Both are available from crates.io. opencv provides a Rust API to the OpenCV library, while ndarray provides multi-dimensional arrays, just as numpy does, along with some mathematical operations (such as calculating mean and standard deviation) on those arrays. ndarray also provides many of the advanced array features of numpy, including slicing, broadcasting, and axis manipulation.

Both opencv and ndarray are excellent crates, but I want to take a moment to single out ndarray for special praise. It is one of my absolute favourite crates: it presents a huge amount of powerful functionality in a very ergonomic API, that combines Rust’s type system with judicious use of macros.

The opencv crate has all of the functionality of the OpenCV library, which (as you know if you’ve ever used OpenCV) is a powerful and sophisticated library. It presents the Mat type for storing images (or, actually, any N-dimensional numeric data).

However, I will argue that it’s better to store your images as ndarray::Array, rather than opencv::Mat, for two reasons:

The Mat type does not work well in Rust. The rank (number of dimensions) of the Mat, and the type of each element, are not part of Mat’s type signature. You need to query the Mat to find out its rank and data type. As far as Rust’s type system is concerned, a Mat which is actually a 2D array of u8 is the same as a Mat that is actually a 3D array of f32. This can lead to confusion over exactly what type of Mat you have (trust me).
ndarray has some numerical processing functionality that you may need. This can only be used on ndarray::Array types, not OpenCV Mats

Historical aside: The need for `BoxedRef`

There used to be a third reason, until v0.89.0 of opencv was released. OpenCV sometimes shares data under the covers. The Mat::roi() function, for example, takes an image and returns a new image that is a smaller subset of the original image. The function looked like this in 0.88.0 of the OpenCV crate:

pub fn roi(m: &Mat, roi: Rect) -> Result<Mat>

Although it returns a new Mat instance, the “new” instance is actually just sharing data with m. You had to read the documentation carefully to discover this fact, and it could lead to hard-to-diagnose bugs. Worse, it could potentially lead to undefined behaviour, since it was possible to end up with two exclusive refs to the same memory.

As of v0.89.0, opencv employs the neat trick of BoxedRef to eliminate this issue. roi() now looks like:

pub fn roi(m: &impl MatTraitConst, roi: Rect) -> Result<BoxedRef<'_, Mat>>

The change to the m parameter is not relevant; MatTraitConst just provides some handy conversions. The important change is BoxedRef. BoxedRef (which, despite its name, does not allocate) ties the lifetime of the returned Mat to m, ensuring that the Rust compiler treats both Mat instances as the same data.

Back to the present: so let’s use `ndarray`

So it’s better to store your image arrays as ndarray::Array instances. Using an ndarray, a 2D array of u8 would be an Array2::<u8>, while a 3D array of f32 would be an Array3::<f32>. Well, actually, the first array would be of type Array<u8, Dim<[Ix; 2]>>, because the rank of an Array is its second generic parameter, but you rarely need to worry about the second parameter. Mostly you can just use Array<n>.

At the time of writing, while rank is part of ndarray::Array’s type signature, the magnitudes of the dimensions are not. An Array2:::<u8> which is 200 x 200 has the same type as an Array2:::<u8> which is 300 x 150. This may change in the future as the Rust compiler improves in the area of generic specialization. (Although whether including dimension magnitudes in the type would actually be more ergonomic is questionable - I have worked with C++ code that used this technique, and it’s definitely a double-edged sword.)

A complication with using ndarray in Rust is that ndarray types and opencv types are different types, and Rust’s strong type system keeps them strictly separate. In Python, the Numpy library’s ndarray type is compatible with OpenCV2’s Mat type, and Python will automagically convert between them. For example, here is a program to load an image, apply some numpy operations, and save back to disk. This program will work as expected:

Listing 1

import cv2
import numpy as np

def normalizeMeanVariance(in_img, mean=(0.485, 0.456, 0.406), variance=(0.229, 0.224, 0.225)):
    img = in_img.copy().astype(np.float32)
    img -= np.array([mean[0] * 255.0, mean[1] * 255.0, mean[2] * 255.0], dtype=np.float32)
    img /= np.array([variance[0] * 255.0, variance[1] * 255.0, variance[2] * 255.0], dtype=np.float32)
    return img

img = cv2.imread("sunflower.jpg", 1)
normalized = normalizeMeanVariance(img)
cv2.imwrite("normalized.jpg", normalized)

The image read by OpenCV (which is a Mat) can be treated as a numpy array for the purpose of numerical calculations, then seamlessly converted back into a Mat to be saved to disk.

In Rust, however, this conversion is not automatic. opencv::Mat and ndarray::Array are two different types, even though they both store N-dimensional arrays of numerical data.

The remainder of this article will describe techniques for achieving the same smooth conversions in Rust.

From `Mat` to `Array`

To convert from a Mat to an ndarray::Array, you can use the various from_shape_xxx() functions in ndarray. For example, here’s how to do it with from_shape_vec():

// Error handling omitted for clarity
fn mat_to_array(m: &opencv::core::Mat) -> ndarray::Array3<u8> {
    const NB_COLOR_CHANNELS: usize = 3;

    assert_eq!(m.typ(), CV_8UC3); // Ensure data type of `m` is 8-bit color triplets

    let pixels = m.data_bytes().unwrap();
    Array3::from_shape_vec(
        (m.rows() as usize, m.cols() as usize, NB_COLOR_CHANNELS),
        pixels.to_vec(),
    )
    .unwrap()
}

This copies the entire array. The copy can actually be avoided, but it’s complicated, and generally not worth doing - see Appendix A. Fortunately, this operation is rarely needed if you are storing all your images as Arrays.

From `Array` to `Mat`

To go the other way, from ndarray::Array to Mat, can be done in a zero-copy way:

// Converts a 3D array, where the 3rd dimension is colour, to a `Mat`.
//
// DataType is a trait provided by the `opencv` crate. It implements certain
// methods for numerical types (such as u8 and f32) that can be the numerical
// data type of a `Mat`.
//
// The lifetime annotations are included for clarity; you can omit them.
fn c3_as_mat<'a, T: DataType>(img: &'a Array3<T>) -> anyhow::Result<BoxedRef<'a, Mat>> {
    let (rows, cols, channels) = img.dim();
    if channels != 3 {
        return Err(anyhow::anyhow!("Expected 3 color channels"));
    }

    let data = img
        .as_slice()
        .ok_or_else(|| anyhow::anyhow!("Image was not contiguous"))?;

    // `T::opencv_type()` returns the type of a single-element array - e.g.,
    // `f32::opencv_type()` returns 5 (which is CV_32FC1). To get the 3-element
    // version, we need to add 16:
    let cv_type = T::opencv_type() + 16;

    // SAFETY: We have confirmed that the array data is triplets of `T`, and
    // there must be "rows * cols" such triplets. The fact that `as_slice()`
    // succeeded confirms that the array data is contiguous and in
    // row-major order. The `BoxedRef` ensures that the Mat we are
    // about to create cannot outlive the array data to which it refers.
    let m: BoxedRef<_> = unsafe {
        Mat::new_rows_cols_with_data_unsafe(
            rows as i32,
            cols as i32,
            cv_type,
            data.as_ptr() as *mut std::ffi::c_void,
            opencv::core::Mat_AUTO_STEP,
        )?
        .into()
    };

    Ok(m)
}

This returns a BoxedRef<Mat> that refers to the same data as img. The BoxedRef<Mat> can be passed into any OpenCV function that expects a &Mat. Thus you can keep your images as Arrays, and temporarily convert them to Mats, with no copying, when needed.

But what about OpenCV functions that return a Mat? Won’t we have to copy them to convert them back into Arrays? No, because OpenCV functions that “return” a Mat almost always take the output Mat as an argument (which is not really suprising, since OpenCV is C++ under the covers, and C++ often works this way), so we can create the mutable equivalent of c3_as_mat(), as follows.

// Returns a `Mat` that is ready to have its data initialized
//
// SAFETY: The data in the returned `Mat` is uninitialized. It must be
// initialized before use.
unsafe fn c3_as_mat_mut_uninit<T: DataType>(
    img: &mut Array3<MaybeUninit<T>>
) -> anyhow::Result<BoxedRefMut<Mat>> {
    let (rows, cols, channels) = img.dim();
    if channels != 3 {
        return Err(anyhow::anyhow!("Expected 3 color channels"));
    }

    let data = img
        .as_slice_mut()
        .ok_or_else(|| anyhow::anyhow!("Image was not contiguous"))?;

    // `T::opencv_type()` returns the type of a single-element array - e.g.,
    // `f32::opencv_type()` returns 5 (which is CV_32FC1). To get the 3-element
    // version, we need to add 16:
    let cv_type = T::opencv_type() + 16;

    // SAFETY: We have confirmed that the array data is (possibly uninitialized)
    // triplets of `T`, and there must be "rows * cols" such triplets. The fact
    // that `as_slice_mut()` succeeded confirms that the array data is contiguous
    // and in row-major order. The `BoxedRefMut` ensures that the Mat we are
    // about to create cannot outlive the array data to which it refers, nor
    // can the data be mutated by any other code.
    let m: BoxedRefMut<_> = unsafe {
        Mat::new_rows_cols_with_data_unsafe(
            rows as i32,
            cols as i32,
            cv_type,
            data.as_mut_ptr() as *mut std::ffi::c_void,
            opencv::core::Mat_AUTO_STEP,
        )?
        .into()
    };

    Ok(m)
}

Note: this function is “unsafe” because it results in a Mat that contains potentially uninitialized data; apart from that, it is completely safe. A Mat with uninitialized data is probably safe - I’ve tried it, and it worked without crashing - but I’ve been unable to confirm that this is definitely allowed with Mat types, so for now, the function has to be considered “unsafe”.

Putting these together, here’s a function that uses OpenCV’s resize() function on images stored as Arrays:

// Resize an image to the given dimensions.
fn resize(img: &Array3<u8>, new_width: usize, new_height: usize) -> anyhow::Result<Array3<u8>> {
    const NB_CHANNELS: usize = 3;

    let mut resized = Array3::<u8>::uninit((new_height, new_width, NB_CHANNELS));

    unsafe {
        // SAFETY: The only unsafe code here is `c3_as_mat_mut_uninit()`, and it is
        // safe as long as the uninitialized `Mat` it returns is subsequently
        // initialized (which resize() does).
        opencv::imgproc::resize(
            &c3_as_mat(img)?,
            &mut c3_as_mat_mut_uninit(&mut resized)?,
            opencv::core::Size::new(new_width as i32, new_height as i32),
            0.0,
            0.0,
            opencv::imgproc::INTER_LINEAR,
        )?;

        // SAFETY: assume_init() is safe if the data is fully initialized, which it
        // now is.
        Ok(resized.assume_init())
    }
}

Finally, here’s the original Python program Listing 1, ported to Rust. You will note that it is considerably longer than the original Python (Python is a concise language, no doubt about that 😊). But here at IDVerse, we have found that as our Rust programs grow in size and complexity, and apply more image processing algorithms, storing our images as Arrays over Mats pays dividends in managing that complexity.

Listing 2

// anyhow   = "1.0.86"
// ndarray  = "0.16.0"
// opencv   = "0.92.2"

use ndarray::{arr3, Array3};
use opencv::{
    boxed_ref::BoxedRef,
    core::{VecN, Vector},
    imgcodecs::{imread, imwrite},
    prelude::*,
};

fn main() {
    let img = imread("sunflower.jpg", 1).unwrap();

    // Convert to our preferred `Array` type, and convert to `f32` at the
    // same time.
    const NB_COLOR_CHANNELS: usize = 3;
    let pixels: Vec<f32> = img
        .data_typed::<VecN<u8, NB_COLOR_CHANNELS>>()
        .unwrap()
        .into_iter()
        .flat_map(|v| [v.0[0], v.0[1], v.0[2]])
        .map(|b| b as f32)
        .collect();
    let img = Array3::from_shape_vec(
        (img.rows() as usize, img.cols() as usize, NB_COLOR_CHANNELS),
        pixels,
    )
    .unwrap();

    // Apply the `ndarray` operation
    let normalized = normalize_mean_variance(&img, (0.485, 0.456, 0.406), (0.229, 0.224, 0.225));

    // Convert back to OpenCV `Mat` and save to disk
    let normalized = c3_as_mat(&normalized).unwrap();
    imwrite("normalized.jpg", &normalized, &Vector::new()).unwrap();
}

fn normalize_mean_variance(
    arr: &Array3<f32>,
    mean: (f32, f32, f32),
    variance: (f32, f32, f32),
) -> Array3<f32> {
    let rhs = arr3(&[[[mean.0 * 255.0, mean.1 * 255.0, mean.2 * 255.0]]]);
    let arr = arr - rhs;

    let rhs = arr3(&[[[variance.0 * 255.0, variance.1 * 255.0, variance.2 * 255.0]]]);
    let arr = arr / rhs;

    arr
}

// Converts a 3D array, where the 3rd dimension is colour, to a `Mat`
fn c3_as_mat<'a, T: DataType>(img: &'a Array3<T>) -> anyhow::Result<BoxedRef<'a, Mat>> {
    let (rows, cols, channels) = img.dim();
    if channels != 3 {
        return Err(anyhow::anyhow!("Expected 3 color channels"));
    }

    let data = img
        .as_slice()
        .ok_or_else(|| anyhow::anyhow!("Image was not contiguous"))?;

    // `T::opencv_type()` returns the type of a single-element array - e.g.,
    // `f32::opencv_type()` returns 5 (which is CV_32FC1). To get the 3-element
    // version, we need to add 16:
    let cv_type = T::opencv_type() + 16;

    // SAFETY: We have confirmed that the array data is triplets of `T`, and
    // there must be "rows * cols" such triplets. The fact that `as_slice()`
    // succeeded confirms that the array data is contiguous and in
    // row-major order. The `BoxedRef` ensures that the Mat we are
    // about to create cannot outlive the array data to which it refers.
    let m: BoxedRef<_> = unsafe {
        Mat::new_rows_cols_with_data_unsafe(
            rows as i32,
            cols as i32,
            cv_type,
            data.as_ptr() as *mut std::ffi::c_void,
            opencv::core::Mat_AUTO_STEP,
        )?
        .into()
    };

    Ok(m)
}

Appendix A

To convert a Mat into an array without copying, you can do this:

fn main() {
    let img = imread("sunflower.jpg", 1).unwrap();

    let ptr = img.data();
    let arr: ArrayView3<u8> =
        unsafe { ArrayView3::from_shape_ptr((img.rows() as usize, img.cols() as usize, 3), ptr) };

    // NOT SAFE: This array shares the data from `img`, so this code is unsafe, since the
    // array could potentially outlive `img`, and `img` could potentially be modified.
}

This works, but to make it safe, you’d have to bind the Mat to the array in some way - maybe put them in a common struct - and ensure that the Mat isn’t dropped or modified as long as the array exists. I experimented with some approaches, but found it more trouble than it’s worth.

Using OpenCV and NdArray in Rust

Introduction

Historical aside: The need for BoxedRef

Back to the present: so let’s use ndarray