gst-plugin-linescan/ROLLINGSUM_GUIDE.md

# GStreamer Rolling Sum Plugin - Complete Documentation

## Table of Contents
- [Overview](#overview)
- [How It Works](#how-it-works)
- [Architecture & Design](#architecture--design)
- [Plugin Properties](#plugin-properties)
- [Basic Usage](#basic-usage)
- [Debugging](#debugging)
- [CSV Analysis](#csv-analysis)
- [Recommended Thresholds](#recommended-thresholds)
- [Troubleshooting](#troubleshooting)
- [Performance](#performance)
- [Integration Examples](#integration-examples)
- [Developer Guide](#developer-guide)
- [References](#references)

## Overview

The `rollingsum` plugin analyzes video frames in real-time by tracking the mean pixel intensity of a specific column across frames. It maintains a rolling window of these values and can drop frames that deviate significantly from the rolling mean, useful for detecting and filtering unstable or anomalous frames.

**Element Name:** `rollingsum` - Transform element that analyzes pixel values and selectively drops frames

**Purpose:** Monitor a vertical column of pixels in video frames, calculate the rolling mean over a time window, and drop frames when the current frame's column mean deviates significantly from the rolling mean baseline.

## How It Works

1. **Column Analysis**: Extracts mean pixel intensity from a specified vertical column
2. **Rolling Window**: Maintains a circular buffer of recent column means
3. **Deviation Detection**: Calculates how much each frame deviates from the rolling mean
4. **Frame Filtering**: Optionally drops frames exceeding the deviation threshold
5. **CSV Logging**: Records all frame statistics for analysis

### Data Flow

```mermaid
graph TD
    A[Video Frame] --> B[Extract Column]
    B --> C[Calculate Column Mean]
    C --> D[Store in Ring Buffer]
    D --> E[Update Rolling Mean]
    E --> F{Deviation > Threshold?}
    F -->|Yes| G[DROP Frame]
    F -->|No| H[PASS Frame]
    C --> E
    
    style G fill:#ff6b6b
    style H fill:#51cf66
```

### Ring Buffer Operation

```mermaid
graph LR
    subgraph Ring Buffer
        A[0] --> B[1]
        B --> C[2]
        C --> D[...]
        D --> E[N-1]
        E ---|wrap| A
    end
    
    F[New Frame Mean] --> G[ring_index]
    G --> A
    
    H[Rolling Mean] --> I[Sum all values]
    I --> J[Divide by count]
    
    style G fill:#ffd43b
```

## Architecture & Design

### Base Class
- Inherits from `GstBaseTransform` (similar to [`select`](gst/select/gstselect.c))
- In-place transform (analysis only, no frame modification)
- Returns `GST_BASE_TRANSFORM_FLOW_DROPPED` to drop frames
- Returns `GST_FLOW_OK` to pass frames

### Element Structure

```c
struct _GstRollingSum
{
  GstBaseTransform element;
  
  /* Properties */
  gint window_size;      // Number of frames in rolling window (default: 1000)
  gint column_index;     // Which column to analyze (default: 1, second column)
  gint stride;           // Row sampling stride (default: 1, every row)
  gdouble threshold;     // Deviation threshold for dropping (default: 0.5)
  gchar *csv_file;       // CSV output file path (default: NULL)
  
  /* State */
  gdouble *ring_buffer;  // Circular buffer of column means
  gint ring_index;       // Current position in ring buffer
  gint frame_count;      // Total frames processed
  gdouble rolling_mean;  // Current rolling mean
  FILE *csv_fp;          // CSV file pointer
};
```

### Algorithm (Simplified from wissotsky's cli.py)

**Per Frame Processing:**

1. **Extract column data:**
   - Select column at column_index
   - Sample every stride rows
   - Calculate mean of sampled pixels: frame_mean
   
2. **Update ring buffer:**
   - Store frame_mean in ring_buffer[ring_index]
   - Increment ring_index (wrap around)
   
3. **Calculate rolling mean:**
   - Sum values in ring buffer (up to window_size or frame_count)
   - Divide by actual window size
   
4. **Calculate deviation:**
   - deviation = abs(frame_mean - rolling_mean)
   - normalized_deviation = deviation / 255.0 (for 8-bit video)
   
5. **Decision:**
   - If normalized_deviation > threshold: DROP frame
   - Else: PASS frame

**Key Simplifications from cli.py:**
- **No EMA tracking**: Use simple rolling mean instead of exponential moving average
- **No variance tracking**: Use fixed threshold instead of dynamic variance-based detection
- **No recording logic**: Just drop/pass, no buffering for output segments
- **No patience mechanism**: Immediate decision per frame

### Video Format Support

**Initial Implementation:**
- **Primary target**: Grayscale (GRAY8, GRAY16)
- **Secondary**: Bayer formats (common in machine vision)

**Caps Filter:**
```c
static GstStaticPadTemplate sink_template =
  GST_STATIC_PAD_TEMPLATE ("sink",
    GST_PAD_SINK,
    GST_PAD_ALWAYS,
    GST_STATIC_CAPS (
      "video/x-raw, format=(string){GRAY8,GRAY16_LE,GRAY16_BE}; "
      "video/x-bayer, format=(string){bggr,grbg,gbrg,rggb}"
    )
  );
```

## Plugin Properties

| Property | Type | Default | Range | Description |
|----------|------|---------|-------|-------------|
| `window-size` | int | 1000 | 1-100000 | Number of frames in rolling window |
| `column-index` | int | 1 | 0-width | Which vertical column to analyze (0-based) |
| `stride` | int | 1 | 1-height | Row sampling stride (1 = every row) |
| `threshold` | double | 0.5 | 0.0-1.0 | Normalized deviation threshold for dropping frames |
| `csv-file` | string | NULL | - | Path to CSV file for logging (NULL = no logging) |

### Understanding Normalized Deviation

- **Range**: 0.0 to 1.0
- **Calculation**: `absolute_deviation / 255.0` (for 8-bit video)
- **Meaning**: Fraction of the full pixel range
  - `0.001` = deviation of ~0.255 pixel values
  - `0.01` = deviation of ~2.55 pixel values
  - `0.1` = deviation of ~25.5 pixel values

## Basic Usage

### Simple Pipeline

```powershell
gst-launch-1.0 idsueyesrc config-file=config.ini ! `
  videoconvert ! `
  video/x-raw,format=GRAY8 ! `
  rollingsum window-size=1000 column-index=1 threshold=0.0002 ! `
  autovideosink
```

### With CSV Logging

```powershell
gst-launch-1.0 idsueyesrc config-file=config.ini exposure=0.5 ! `
  videoconvert ! `
  video/x-raw,format=GRAY8 ! `
  rollingsum window-size=1000 column-index=1 threshold=0.0002 csv-file=output.csv ! `
  fakesink
```

### Custom Configuration

```powershell
gst-launch-1.0 idsueyesrc config-file=config.ini ! `
  rollingsum window-size=5000 column-index=320 stride=2 threshold=0.3 ! `
  queue ! `
  autovideosink
```

### With Format Conversion

```powershell
gst-launch-1.0 idsueyesrc ! `
  videoconvert ! `
  video/x-raw,format=GRAY8 ! `
  rollingsum ! `
  autovideosink
```

## Debugging

### Enable Debug Output

Use the `GST_DEBUG` environment variable to see detailed plugin operation:

#### Windows PowerShell

```powershell
$env:GST_DEBUG="rollingsum:5"; gst-launch-1.0 [pipeline...]
```

#### Windows CMD

```cmd
set GST_DEBUG=rollingsum:5 && gst-launch-1.0 [pipeline...]
```

#### Linux/Mac

```bash
GST_DEBUG=rollingsum:5 gst-launch-1.0 [pipeline...]
```

### Debug Levels

| Level | Output |
|-------|--------|
| `rollingsum:1` | Errors only |
| `rollingsum:2` | Warnings |
| `rollingsum:3` | Info messages (file open/close) |
| `rollingsum:4` | Debug (caps negotiation) |
| `rollingsum:5` | Log (all frame processing) |

### Example Debug Output

```
0:00:04.029432200 DEBUG rollingsum gstrollingsum.c:436: Extracted column mean: 10.07
0:00:04.032257100 DEBUG rollingsum gstrollingsum.c:466: Frame 1: mean=10.07, rolling_mean=10.07, deviation=0.00 (normalized=0.0000)
```

**Key Fields:**
- `Frame N`: Frame number
- `mean`: Current frame's column mean
- `rolling_mean`: Average of last N frames (window-size)
- `deviation`: Absolute difference
- `normalized`: Deviation as fraction of 255

### Common Debug Scenarios

#### 1. Verify Plugin Loaded

```powershell
gst-inspect-1.0 rollingsum
```

Should show plugin details. If not found, check `GST_PLUGIN_PATH`.

#### 2. Check CSV File Creation

Look for this in debug output:
```
INFO rollingsum: Opened CSV file: output.csv
```

#### 3. Monitor Frame Drops

Look for:
```
DEBUG rollingsum: Dropping frame 42: deviation 0.0005 > threshold 0.0002
```

#### 4. Verify Caps Negotiation

```
DEBUG rollingsum: set_caps
DEBUG rollingsum: Video format: GRAY8, 1224x1026
```

## CSV Analysis

### CSV Format

The output CSV contains:

```csv
frame,column_mean,rolling_mean,deviation,normalized_deviation,dropped
1,10.071150,10.071150,0.000000,0.000000,0
2,10.059454,10.065302,0.005848,0.000023,0
...
```

### Analyze Results

Use the included analysis script:

```powershell
uv run scripts/analyze_sma.py output.csv
```

**Output includes:**
- Statistical summary (min/max/mean/std)
- Threshold recommendations based on percentiles
- Standard deviation-based suggestions
- Visualization plots saved to `results/debug/`
- Archived CSV with timestamp in `results/debug/`

**Output files are automatically organized:**
- `results/debug/output_YYYYMMDD_HHMMSS.csv` - Archived CSV
- `results/debug/output_analysis_YYYYMMDD_HHMMSS.png` - Analysis plots

The `results/` directory is gitignored to keep your repository clean.

### Interpreting Results

The analysis provides threshold recommendations:

| Percentile | Description | Use Case |
|------------|-------------|----------|
| 99th | Drops top 1% | Very conservative, catch only extreme outliers |
| 95th | Drops top 5% | Conservative, good for quality control |
| 90th | Drops top 10% | Balanced, moderate filtering |
| 75th | Drops top 25% | Aggressive, maximum quality |

## Recommended Thresholds

Based on analysis of stable camera footage:

### For General Use

```powershell
# Conservative (1-2% frame drop)
threshold=0.0003

# Moderate (5-10% frame drop)
threshold=0.0002

# Aggressive (20-25% frame drop)
threshold=0.0001
```

### For Specific Scenarios

**High-speed acquisition** (minimal processing):
```powershell
window-size=100 threshold=0.0005
```

**Quality-focused** (stable scenes):
```powershell
window-size=1000 threshold=0.0001
```

**Real-time monitoring** (fast response):
```powershell
window-size=50 threshold=0.0002
```

## Troubleshooting

### No frames being dropped (threshold too high)

**Symptom**: `dropped` column always 0 in CSV

**Solution**:
1. Run with CSV logging
2. Analyze with `uv run scripts/analyze_sma.py output.csv`
3. Use recommended threshold from 90th-99th percentile

### Too many frames dropped (threshold too low)

**Symptom**: Most frames have `dropped=1`, choppy video

**Solution**:
1. Increase threshold (try doubling current value)
2. Check if column_index is appropriate
3. Verify video is stable (not shaking/moving)

### CSV file not created

**Check**:
1. File path is writable
2. Look for "Opened CSV file" in debug output (`GST_DEBUG=rollingsum:3`)
3. Verify csv-file property is set correctly

### Column index out of range

**Symptom**:
```
WARNING rollingsum: Column index 1000 >= width 1224, using column 0
```

**Solution**: Set `column-index` to value < video width

### Inconsistent results

**Possible causes**:
1. Window size too small (< 50 frames)
2. Sampling moving/dynamic content
3. Column contains edge/artifact data

**Solutions**:
- Increase `window-size` to 500-1000
- Choose different `column-index` (avoid edges)
- Use `stride=2` or higher for faster processing

## Performance

### Performance Tips

1. **Larger window = more stable** but slower to adapt to scene changes
2. **Stride > 1** reduces computation but less accurate column mean
3. **CSV logging** has minimal performance impact
4. **Debug level 5** can produce massive logs, use only when needed

### Memory Usage

- Ring buffer: `window_size * sizeof(double)` = ~8KB for default 1000 frames
- Minimal per-frame allocation

### CPU Usage

- Column extraction: O(height/stride)
- Rolling mean update: O(1) using incremental sum
- Very lightweight compared to full-frame processing

### Optimization Opportunities

1. **Incremental mean**: Track sum instead of recalculating
2. **SIMD**: Vectorize column summation
3. **Skip calculation**: Only recalc every N frames if baseline is stable

## Integration Examples

### Python Script Control

```python
import subprocess

# Run pipeline with CSV logging
subprocess.run([
    'gst-launch-1.0',
    'idsueyesrc', 'config-file=config.ini',
    '!', 'videoconvert',
    '!', 'video/x-raw,format=GRAY8',
    '!', 'rollingsum',
        'window-size=1000',
        'column-index=1',
        'threshold=0.0002',
        'csv-file=output.csv',
    '!', 'fakesink'
])

# Analyze results
subprocess.run(['uv', 'run', 'scripts/analyze_sma.py', 'output.csv'])
```

### Adaptive Threshold

Use analysis results to set optimal threshold for next run:

```python
import pandas as pd

# Analyze previous run
df = pd.read_csv('output.csv')
recommended_threshold = df['normalized_deviation'].quantile(0.95)

print(f"Recommended threshold: {recommended_threshold:.6f}")
```

## Developer Guide

### Implementation Files

**Directory Structure:**
```
gst/rollingsum/
├── CMakeLists.txt
├── gstrollingsum.c
└── gstrollingsum.h
```

**gstrollingsum.h:**
- Element type definitions
- Structure declarations
- Property enums
- Function prototypes

**gstrollingsum.c:**
- GObject methods (init, dispose, get/set properties)
- GstBaseTransform methods (transform_ip)
- Helper functions (extract_column_mean, update_rolling_mean)
- Plugin registration

**CMakeLists.txt:**
- Build configuration (copy from [`gst/select/CMakeLists.txt`](gst/select/CMakeLists.txt))
- Link GStreamer base and video libraries

### Adding New Features

Key files:
- [`gst/rollingsum/gstrollingsum.c`](gst/rollingsum/gstrollingsum.c) - Main implementation
- [`gst/rollingsum/gstrollingsum.h`](gst/rollingsum/gstrollingsum.h) - Header/structures
- [`gst/rollingsum/CMakeLists.txt`](gst/rollingsum/CMakeLists.txt) - Build config

### Rebuild After Changes

```powershell
.\build.ps1  # Windows
```

```bash
./build.sh   # Linux
```

### Testing

```powershell
# Quick test
gst-inspect-1.0 rollingsum

# Full pipeline test with debug
$env:GST_DEBUG="rollingsum:5"
gst-launch-1.0 videotestsrc ! rollingsum ! fakesink
```

### Testing Strategy

**Unit Tests:**
- Ring buffer wrapping
- Mean calculation accuracy
- Threshold comparison logic

**Integration Tests:**
- Pipeline with videotestsrc
- Pipeline with idsueyesrc
- Frame drop verification
- Property changes during playback

**Test Cases:**
1. Static video (all frames similar) → all pass
2. Single bright frame → that frame drops
3. Gradual change → frames pass
4. Periodic pattern → pattern frames drop

### Integration with Existing Project

**Build System:**
Update [`gst/CMakeLists.txt`](gst/CMakeLists.txt):
```cmake
add_subdirectory (rollingsum)
```

**Documentation:**
Update [`README.md`](README.md):
- Add rollingsum to "Other elements" section
- Add pipeline example

### Future Enhancements

**Phase 2 (If Needed):**
- Add EMA baseline tracking (like cli.py)
- Add variance-based thresholds
- Support multiple columns or regions
- Add metadata output (tag frames with deviation values)
- RGB format support (analyze specific channel)

**Phase 3 (Advanced):**
- Full cli.py recording logic
- Buffer and output segments
- Integration with probe detection systems

### Implementation Checklist

- [x] Create gst/rollingsum directory
- [x] Implement gstrollingsum.h
- [x] Implement gstrollingsum.c
- [x] Create CMakeLists.txt
- [x] Update gst/CMakeLists.txt
- [x] Build and test basic functionality
- [x] Test with idsueyesrc
- [x] Update README.md
- [x] Create feature branch
- [x] Commit and document

## References

- Original algorithm: `cli.py` lines 64-79 (column extraction and mean comparison)
- Template element: [`gst/select/gstselect.c`](gst/select/gstselect.c)
- GStreamer base transform: [GstBaseTransform documentation](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html)
- [scripts/analyze_sma.py](scripts/analyze_sma.py) - Analysis tool
- GStreamer documentation: https://gstreamer.freedesktop.org/documentation/

## Support

For issues or questions:
1. Enable debug output (`$env:GST_DEBUG="rollingsum:5"` in PowerShell)
2. Generate CSV log and analyze
3. Check this guide's troubleshooting section
4. Review debug output for errors/warnings