gst-plugin-linescan/ROLLINGSUM_GUIDE.md

GStreamer Rolling Sum Plugin - Complete Documentation

Table of Contents

• Overview
• How It Works
• Architecture & Design
• Plugin Properties
• Basic Usage
• Debugging
• CSV Analysis
• Recommended Thresholds
• Troubleshooting
• Performance
• Integration Examples
• Developer Guide
• References

Overview

The rollingsum plugin analyzes video frames in real-time by tracking the mean pixel intensity of a specific column across frames. It maintains a rolling window of these values and can drop frames that deviate significantly from the rolling mean, useful for detecting and filtering unstable or anomalous frames.

Element Name: rollingsum - Transform element that analyzes pixel values and selectively drops frames

Purpose: Monitor a vertical column of pixels in video frames, calculate the rolling mean over a time window, and drop frames when the current frame's column mean deviates significantly from the rolling mean baseline.

How It Works

1. Column Analysis: Extracts mean pixel intensity from a specified vertical column
2. Rolling Window: Maintains a circular buffer of recent column means
3. Deviation Detection: Calculates how much each frame deviates from the rolling mean
4. Frame Filtering: Optionally drops frames exceeding the deviation threshold
5. CSV Logging: Records all frame statistics for analysis

Data Flow

graph TD
    A[Video Frame] --> B[Extract Column]
    B --> C[Calculate Column Mean]
    C --> D[Store in Ring Buffer]
    D --> E[Update Rolling Mean]
    E --> F{Deviation > Threshold?}
    F -->|Yes| G[DROP Frame]
    F -->|No| H[PASS Frame]
    C --> E
    
    style G fill:#ff6b6b
    style H fill:#51cf66

Ring Buffer Operation

graph LR
    subgraph Ring Buffer
        A[0] --> B[1]
        B --> C[2]
        C --> D[...]
        D --> E[N-1]
        E ---|wrap| A
    end
    
    F[New Frame Mean] --> G[ring_index]
    G --> A
    
    H[Rolling Mean] --> I[Sum all values]
    I --> J[Divide by count]
    
    style G fill:#ffd43b

Architecture & Design

Base Class

• Inherits from GstBaseTransform (similar to select)
• In-place transform (analysis only, no frame modification)
• Returns GST_BASE_TRANSFORM_FLOW_DROPPED to drop frames
• Returns GST_FLOW_OK to pass frames

Element Structure

struct _GstRollingSum
{
  GstBaseTransform element;
  
  /* Properties */
  gint window_size;      // Number of frames in rolling window (default: 1000)
  gint column_index;     // Which column to analyze (default: 1, second column)
  gint stride;           // Row sampling stride (default: 1, every row)
  gdouble threshold;     // Deviation threshold for dropping (default: 0.5)
  gchar *csv_file;       // CSV output file path (default: NULL)
  
  /* State */
  gdouble *ring_buffer;  // Circular buffer of column means
  gint ring_index;       // Current position in ring buffer
  gint frame_count;      // Total frames processed
  gdouble rolling_mean;  // Current rolling mean
  FILE *csv_fp;          // CSV file pointer
};

Algorithm (Simplified from cli.py)

Per Frame Processing:

1. Extract column data:

   • Select column at column_index
   • Sample every stride rows
   • Calculate mean of sampled pixels: frame_mean

2. Update ring buffer:

   • Store frame_mean in ring_buffer[ring_index]
   • Increment ring_index (wrap around)

3. Calculate rolling mean:

   • Sum values in ring buffer (up to window_size or frame_count)
   • Divide by actual window size

4. Calculate deviation:

   • deviation = abs(frame_mean - rolling_mean)
   • normalized_deviation = deviation / 255.0 (for 8-bit video)

5. Decision:

   • If normalized_deviation > threshold: DROP frame
   • Else: PASS frame

Key Simplifications from cli.py:

• No EMA tracking: Use simple rolling mean instead of exponential moving average
• No variance tracking: Use fixed threshold instead of dynamic variance-based detection
• No recording logic: Just drop/pass, no buffering for output segments
• No patience mechanism: Immediate decision per frame

Video Format Support

Initial Implementation:

• Primary target: Grayscale (GRAY8, GRAY16)
• Secondary: Bayer formats (common in machine vision)

Caps Filter:

static GstStaticPadTemplate sink_template =
  GST_STATIC_PAD_TEMPLATE ("sink",
    GST_PAD_SINK,
    GST_PAD_ALWAYS,
    GST_STATIC_CAPS (
      "video/x-raw, format=(string){GRAY8,GRAY16_LE,GRAY16_BE}; "
      "video/x-bayer, format=(string){bggr,grbg,gbrg,rggb}"
    )
  );

Plugin Properties

Property	Type	Default	Range	Description
window-size	int	1000	1-100000	Number of frames in rolling window
column-index	int	1	0-width	Which vertical column to analyze (0-based)
stride	int	1	1-height	Row sampling stride (1 = every row)
threshold	double	0.5	0.0-1.0	Normalized deviation threshold for dropping frames
csv-file	string	NULL	-	Path to CSV file for logging (NULL = no logging)

Understanding Normalized Deviation

• Range: 0.0 to 1.0
• Calculation: absolute_deviation / 255.0 (for 8-bit video)
• Meaning: Fraction of the full pixel range
  • 0.001 = deviation of ~0.255 pixel values
  • 0.01 = deviation of ~2.55 pixel values
  • 0.1 = deviation of ~25.5 pixel values

Basic Usage

Simple Pipeline

gst-launch-1.0 idsueyesrc config-file=config.ini ! `
  videoconvert ! `
  video/x-raw,format=GRAY8 ! `
  rollingsum window-size=1000 column-index=1 threshold=0.0002 ! `
  autovideosink

With CSV Logging

gst-launch-1.0 idsueyesrc config-file=config.ini exposure=0.5 ! `
  videoconvert ! `
  video/x-raw,format=GRAY8 ! `
  rollingsum window-size=1000 column-index=1 threshold=0.0002 csv-file=output.csv ! `
  fakesink

Custom Configuration

gst-launch-1.0 idsueyesrc config-file=config.ini ! `
  rollingsum window-size=5000 column-index=320 stride=2 threshold=0.3 ! `
  queue ! `
  autovideosink

With Format Conversion

gst-launch-1.0 idsueyesrc ! `
  videoconvert ! `
  video/x-raw,format=GRAY8 ! `
  rollingsum ! `
  autovideosink

Debugging

Enable Debug Output

Use the GST_DEBUG environment variable to see detailed plugin operation:

Windows PowerShell

$env:GST_DEBUG="rollingsum:5"; gst-launch-1.0 [pipeline...]

Windows CMD

set GST_DEBUG=rollingsum:5 && gst-launch-1.0 [pipeline...]

Linux/Mac

GST_DEBUG=rollingsum:5 gst-launch-1.0 [pipeline...]

Debug Levels

Level	Output
rollingsum:1	Errors only
rollingsum:2	Warnings
rollingsum:3	Info messages (file open/close)
rollingsum:4	Debug (caps negotiation)
rollingsum:5	Log (all frame processing)

Example Debug Output

0:00:04.029432200 DEBUG rollingsum gstrollingsum.c:436: Extracted column mean: 10.07
0:00:04.032257100 DEBUG rollingsum gstrollingsum.c:466: Frame 1: mean=10.07, rolling_mean=10.07, deviation=0.00 (normalized=0.0000)

Key Fields:

• Frame N: Frame number
• mean: Current frame's column mean
• rolling_mean: Average of last N frames (window-size)
• deviation: Absolute difference
• normalized: Deviation as fraction of 255

Common Debug Scenarios

1. Verify Plugin Loaded

gst-inspect-1.0 rollingsum

Should show plugin details. If not found, check GST_PLUGIN_PATH.

2. Check CSV File Creation

Look for this in debug output:

INFO rollingsum: Opened CSV file: output.csv

3. Monitor Frame Drops

Look for:

DEBUG rollingsum: Dropping frame 42: deviation 0.0005 > threshold 0.0002

4. Verify Caps Negotiation

DEBUG rollingsum: set_caps
DEBUG rollingsum: Video format: GRAY8, 1224x1026

CSV Analysis

CSV Format

The output CSV contains:

frame,column_mean,rolling_mean,deviation,normalized_deviation,dropped
1,10.071150,10.071150,0.000000,0.000000,0
2,10.059454,10.065302,0.005848,0.000023,0
...

Analyze Results

Use the included analysis script:

uv run scripts/analyze_sma.py output.csv

Output includes:

• Statistical summary (min/max/mean/std)
• Threshold recommendations based on percentiles
• Standard deviation-based suggestions
• Visualization plots saved to results/debug/
• Archived CSV with timestamp in results/debug/

Output files are automatically organized:

• results/debug/output_YYYYMMDD_HHMMSS.csv - Archived CSV
• results/debug/output_analysis_YYYYMMDD_HHMMSS.png - Analysis plots

The results/ directory is gitignored to keep your repository clean.

Interpreting Results

The analysis provides threshold recommendations:

Percentile	Description	Use Case
99th	Drops top 1%	Very conservative, catch only extreme outliers
95th	Drops top 5%	Conservative, good for quality control
90th	Drops top 10%	Balanced, moderate filtering
75th	Drops top 25%	Aggressive, maximum quality

Recommended Thresholds

Based on analysis of stable camera footage:

For General Use

# Conservative (1-2% frame drop)
threshold=0.0003

# Moderate (5-10% frame drop)
threshold=0.0002

# Aggressive (20-25% frame drop)
threshold=0.0001

For Specific Scenarios

High-speed acquisition (minimal processing):

window-size=100 threshold=0.0005

Quality-focused (stable scenes):

window-size=1000 threshold=0.0001

Real-time monitoring (fast response):

window-size=50 threshold=0.0002

Troubleshooting

No frames being dropped (threshold too high)

Symptom: dropped column always 0 in CSV

Solution:

1. Run with CSV logging
2. Analyze with uv run scripts/analyze_sma.py output.csv
3. Use recommended threshold from 90th-99th percentile

Too many frames dropped (threshold too low)

Symptom: Most frames have dropped=1, choppy video

Solution:

1. Increase threshold (try doubling current value)
2. Check if column_index is appropriate
3. Verify video is stable (not shaking/moving)

CSV file not created

Check:

1. File path is writable
2. Look for "Opened CSV file" in debug output (GST_DEBUG=rollingsum:3)
3. Verify csv-file property is set correctly

Column index out of range

Symptom:

WARNING rollingsum: Column index 1000 >= width 1224, using column 0

Solution: Set column-index to value < video width

Inconsistent results

Possible causes:

1. Window size too small (< 50 frames)
2. Sampling moving/dynamic content
3. Column contains edge/artifact data

Solutions:

• Increase window-size to 500-1000
• Choose different column-index (avoid edges)
• Use stride=2 or higher for faster processing

Performance

Performance Tips

1. Larger window = more stable but slower to adapt to scene changes
2. Stride > 1 reduces computation but less accurate column mean
3. CSV logging has minimal performance impact
4. Debug level 5 can produce massive logs, use only when needed

Memory Usage

• Ring buffer: window_size * sizeof(double) = ~8KB for default 1000 frames
• Minimal per-frame allocation

CPU Usage

• Column extraction: O(height/stride)
• Rolling mean update: O(1) using incremental sum
• Very lightweight compared to full-frame processing

Optimization Opportunities

1. Incremental mean: Track sum instead of recalculating
2. SIMD: Vectorize column summation
3. Skip calculation: Only recalc every N frames if baseline is stable

Integration Examples

Python Script Control

import subprocess

# Run pipeline with CSV logging
subprocess.run([
    'gst-launch-1.0',
    'idsueyesrc', 'config-file=config.ini',
    '!', 'videoconvert',
    '!', 'video/x-raw,format=GRAY8',
    '!', 'rollingsum',
        'window-size=1000',
        'column-index=1',
        'threshold=0.0002',
        'csv-file=output.csv',
    '!', 'fakesink'
])

# Analyze results
subprocess.run(['uv', 'run', 'scripts/analyze_sma.py', 'output.csv'])

Adaptive Threshold

Use analysis results to set optimal threshold for next run:

import pandas as pd

# Analyze previous run
df = pd.read_csv('output.csv')
recommended_threshold = df['normalized_deviation'].quantile(0.95)

print(f"Recommended threshold: {recommended_threshold:.6f}")

Developer Guide

Implementation Files

Directory Structure:

gst/rollingsum/
├── CMakeLists.txt
├── gstrollingsum.c
└── gstrollingsum.h

gstrollingsum.h:

• Element type definitions
• Structure declarations
• Property enums
• Function prototypes

gstrollingsum.c:

• GObject methods (init, dispose, get/set properties)
• GstBaseTransform methods (transform_ip)
• Helper functions (extract_column_mean, update_rolling_mean)
• Plugin registration

CMakeLists.txt:

• Build configuration (copy from gst/select/CMakeLists.txt)
• Link GStreamer base and video libraries

Adding New Features

Key files:

• gst/rollingsum/gstrollingsum.c - Main implementation
• gst/rollingsum/gstrollingsum.h - Header/structures
• gst/rollingsum/CMakeLists.txt - Build config

Rebuild After Changes

.\build.ps1  # Windows

./build.sh   # Linux

Testing

# Quick test
gst-inspect-1.0 rollingsum

# Full pipeline test with debug
$env:GST_DEBUG="rollingsum:5"
gst-launch-1.0 videotestsrc ! rollingsum ! fakesink

Testing Strategy

Unit Tests:

• Ring buffer wrapping
• Mean calculation accuracy
• Threshold comparison logic

Integration Tests:

• Pipeline with videotestsrc
• Pipeline with idsueyesrc
• Frame drop verification
• Property changes during playback

Test Cases:

1. Static video (all frames similar) → all pass
2. Single bright frame → that frame drops
3. Gradual change → frames pass
4. Periodic pattern → pattern frames drop

Integration with Existing Project

Build System: Update gst/CMakeLists.txt:

add_subdirectory (rollingsum)

Documentation: Update README.md:

• Add rollingsum to "Other elements" section
• Add pipeline example

Future Enhancements

Phase 2 (If Needed):

• Add EMA baseline tracking (like cli.py)
• Add variance-based thresholds
• Support multiple columns or regions
• Add metadata output (tag frames with deviation values)
• RGB format support (analyze specific channel)

Phase 3 (Advanced):

• Full cli.py recording logic
• Buffer and output segments
• Integration with probe detection systems

Implementation Checklist

• Create gst/rollingsum directory
• Implement gstrollingsum.h
• Implement gstrollingsum.c
• Create CMakeLists.txt
• Update gst/CMakeLists.txt
• Build and test basic functionality
• Test with idsueyesrc
• Update README.md
• Create feature branch
• Commit and document

References

• Original algorithm: cli.py lines 64-79 (column extraction and mean comparison)
• Template element: gst/select/gstselect.c
• GStreamer base transform: GstBaseTransform documentation
• scripts/analyze_sma.py - Analysis tool
• GStreamer documentation: https://gstreamer.freedesktop.org/documentation/

Support

For issues or questions:

1. Enable debug output ($env:GST_DEBUG="rollingsum:5" in PowerShell)
2. Generate CSV log and analyze
3. Check this guide's troubleshooting section
4. Review debug output for errors/warnings