How the Vision Pipeline Actually Works

Sproutcast's health metrics come from a set of C++ pipelines that run on every captured image. Here is what each pipeline does, without the maths.

The segmentation pipeline (P3b.0)

Before any measurement can happen, the pipeline needs to know which pixels are plant and which are background. P3b.0 does this with HSV colour thresholding — it converts the image from RGB to HSV (Hue, Saturation, Value) and keeps only the pixels that fall within the green-yellow-brown range where foliage lives.

It then applies a few cleanup steps: morphological opening (removes small noise specks), closing (fills gaps in the leaf mask), and a connected-components filter to keep only the largest region. The result is a binary mask — white pixels are plant, black pixels are everything else.

The mask is the foundation for every other pipeline.

The chlorophyll index pipeline (P4.0)

With the mask in hand, P4.0 computes the Excess Green index:

ExG = 2G − R − B

where G, R, B are the normalised mean channel values within the masked region. ExG correlates with chlorophyll content because chlorophyll absorbs red and blue light and reflects green. A healthy leaf has high ExG; a nitrogen-deficient or stressed leaf has lower ExG as chlorophyll degrades.

We normalise ExG to a 0–1 scale based on the expected range for common houseplant species, so the number is interpretable without a reference chart.

The turgor proxy (built into P3b.0)

Turgor pressure — the hydraulic stiffness of a leaf — affects how crisp its edges are. A well-hydrated leaf has stiff cell walls and sharp, clearly-defined edges. A dehydrated leaf has softer edges as cells lose water and the cuticle becomes less taut.

The pipeline measures this by computing the Laplacian variance of the leaf mask boundary. High variance = sharp edges = high turgor. Low variance = blurry edges = potential water stress.

This is a proxy, not a direct measurement. It works well for broad-leaf plants (Monstera, Pothos, most herbs). It is less reliable for needled or waxy-leaved plants where edge sharpness doesn't change much with hydration.

The growth rate pipeline (P6.0)

P6.0 simply computes the change in plant area (% of frame) between the current capture and the one from seven days ago. It divides by the number of days to get a daily growth rate.

To be useful, it needs:

1. Consistent camera framing (same position every capture)

2. A planted_date set on the plant (to compute cumulative growth from day zero)

3. At least two captures, seven days apart

It does not model growth mathematically — it just subtracts two area values and divides. Simple, but it catches the most common problem: a plant that has stopped growing entirely.

How pipelines are added

Every pipeline is a single .cpp file in edge/src/pipelines/ or edge/src/custom/. It implements a simple interface (run an image, return a JSON metrics object) and registers itself with a macro. CMake auto-discovers it — no build system changes needed. This is how we can ship new pipelines as community contributions without touching the core.