diff --git a/DESIGN_ROLLINGSUM.md b/DESIGN_ROLLINGSUM.md deleted file mode 100644 index be391de..0000000 --- a/DESIGN_ROLLINGSUM.md +++ /dev/null @@ -1,290 +0,0 @@ -# Rolling Sum Filter - Design Document - -## Overview - -A GStreamer element that drops frames based on rolling mean analysis of a single column of pixels. Inspired by the detection algorithm in `cli.py`, simplified for real-time streaming. - -## Element Name - -**`rollingsum`** - Transform element that analyzes pixel values and selectively drops frames - -## Purpose - -Monitor a vertical column of pixels in video frames, calculate the rolling mean over a time window, and drop frames when the current frame's column mean deviates significantly from the rolling mean baseline. - -## Architecture - -### Base Class -- Inherits from `GstBaseTransform` (similar to [`select`](gst/select/gstselect.c)) -- In-place transform (analysis only, no frame modification) -- Returns `GST_BASE_TRANSFORM_FLOW_DROPPED` to drop frames -- Returns `GST_FLOW_OK` to pass frames - -### Element Structure - -```c -struct _GstRollingSum -{ - GstBaseTransform element; - - /* Properties */ - gint window_size; // Number of frames in rolling window (default: 1000) - gint column_index; // Which column to analyze (default: 1, second column) - gint stride; // Row sampling stride (default: 1, every row) - gdouble threshold; // Deviation threshold for dropping (default: 0.5) - - /* State */ - gdouble *ring_buffer; // Circular buffer of column means - gint ring_index; // Current position in ring buffer - gint frame_count; // Total frames processed - gdouble rolling_mean; // Current rolling mean -}; -``` - -## Algorithm (Simplified from cli.py) - -### Per Frame Processing - -``` -1. Extract column data: - - Select column at column_index - - Sample every stride rows - - Calculate mean of sampled pixels: frame_mean - -2. Update ring buffer: - - Store frame_mean in ring_buffer[ring_index] - - Increment ring_index (wrap around) - -3. Calculate rolling mean: - - Sum values in ring buffer (up to window_size or frame_count) - - Divide by actual window size - -4. Calculate deviation: - - deviation = abs(frame_mean - rolling_mean) - -5. Decision: - - If deviation > threshold: DROP frame - - Else: PASS frame -``` - -### Key Simplifications from cli.py - -- **No EMA tracking**: Use simple rolling mean instead of exponential moving average -- **No variance tracking**: Use fixed threshold instead of dynamic variance-based detection -- **No recording logic**: Just drop/pass, no buffering for output segments -- **No patience mechanism**: Immediate decision per frame - -## Properties - -| Property | Type | Default | Range | Description | -|----------|------|---------|-------|-------------| -| `window-size` | int | 1000 | 1-100000 | Number of frames in rolling window | -| `column-index` | int | 1 | 0-width | Which vertical column to analyze | -| `stride` | int | 1 | 1-height | Row sampling stride (1=all rows) | -| `threshold` | double | 0.5 | 0.0-1.0 | Normalized deviation threshold for dropping | - -## Data Structures - -### Ring Buffer -- Array of doubles sized to `window_size` -- Stores column mean for each processed frame -- Circular wrapping via modulo: `ring_index % window_size` - -### Frame Analysis -``` -For column_index=1, stride=1, frame height=H: -- Extract pixels: frame[:, column_index] -- Sample: frame[::stride, column_index] -- Count: H / stride samples -- Mean: sum(samples) / count -``` - -## Implementation Files - -### Directory Structure -``` -gst/rollingsum/ -├── CMakeLists.txt -├── gstrollingsum.c -└── gstrollingsum.h -``` - -### gstrollingsum.h -- Element type definitions -- Structure declarations -- Property enums -- Function prototypes - -### gstrollingsum.c -- GObject methods (init, dispose, get/set properties) -- GstBaseTransform methods (transform_ip) -- Helper functions (extract_column_mean, update_rolling_mean) -- Plugin registration - -### CMakeLists.txt -- Build configuration (copy from [`gst/select/CMakeLists.txt`](gst/select/CMakeLists.txt)) -- Link GStreamer base and video libraries - -## Video Format Support - -### Initial Implementation -- **Primary target**: Grayscale (GRAY8, GRAY16) -- **Secondary**: Bayer formats (common in machine vision) - -### Caps Filter -```c -static GstStaticPadTemplate sink_template = - GST_STATIC_PAD_TEMPLATE ("sink", - GST_PAD_SINK, - GST_PAD_ALWAYS, - GST_STATIC_CAPS ( - "video/x-raw, format=(string){GRAY8,GRAY16_LE,GRAY16_BE}; " - "video/x-bayer, format=(string){bggr,grbg,gbrg,rggb}" - ) - ); -``` - -## Pipeline Usage Examples - -### Basic Usage -```bash -gst-launch-1.0 idsueyesrc config-file=config.ini ! \ - rollingsum window-size=1000 column-index=1 threshold=0.5 ! \ - autovideosink -``` - -### Custom Configuration -```bash -gst-launch-1.0 idsueyesrc config-file=config.ini ! \ - rollingsum window-size=5000 column-index=320 stride=2 threshold=0.3 ! \ - queue ! \ - autovideosink -``` - -### With Format Conversion -```bash -gst-launch-1.0 idsueyesrc ! \ - videoconvert ! \ - video/x-raw,format=GRAY8 ! \ - rollingsum ! \ - autovideosink -``` - -## Performance Considerations - -### Memory Usage -- Ring buffer: `window_size * sizeof(double)` = ~8KB for default 1000 frames -- Minimal per-frame allocation - -### CPU Usage -- Column extraction: O(height/stride) -- Rolling mean update: O(1) using incremental sum -- Very lightweight compared to full-frame processing - -### Optimization Opportunities -1. **Incremental mean**: Track sum instead of recalculating -2. **SIMD**: Vectorize column summation -3. **Skip calculation**: Only recalc every N frames if baseline is stable - -## Testing Strategy - -### Unit Tests -- Ring buffer wrapping -- Mean calculation accuracy -- Threshold comparison logic - -### Integration Tests -- Pipeline with videotestsrc -- Pipeline with idsueyesrc -- Frame drop verification -- Property changes during playback - -### Test Cases -1. Static video (all frames similar) → all pass -2. Single bright frame → that frame drops -3. Gradual change → frames pass -4. Periodic pattern → pattern frames drop - -## Future Enhancements - -### Phase 2 (If Needed) -- Add EMA baseline tracking (like cli.py) -- Add variance-based thresholds -- Support multiple columns or regions -- Add metadata output (tag frames with deviation values) -- RGB format support (analyze specific channel) - -### Phase 3 (Advanced) -- Full cli.py recording logic -- Buffer and output segments -- Integration with probe detection systems - -## Integration with Existing Project - -### Build System -Update [`gst/CMakeLists.txt`](gst/CMakeLists.txt): -```cmake -add_subdirectory (rollingsum) -``` - -### Documentation -Update [`README.md`](README.md): -- Add rollingsum to "Other elements" section -- Add pipeline example - -## Mermaid Diagram: Data Flow - -```mermaid -graph TD - A[Video Frame] --> B[Extract Column] - B --> C[Calculate Column Mean] - C --> D[Store in Ring Buffer] - D --> E[Update Rolling Mean] - E --> F{Deviation > Threshold?} - F -->|Yes| G[DROP Frame] - F -->|No| H[PASS Frame] - C --> E - - style G fill:#ff6b6b - style H fill:#51cf66 -``` - -## Mermaid Diagram: Ring Buffer Operation - -```mermaid -graph LR - subgraph Ring Buffer - A[0] --> B[1] - B --> C[2] - C --> D[...] - D --> E[N-1] - E ---|wrap| A - end - - F[New Frame Mean] --> G[ring_index] - G --> A - - H[Rolling Mean] --> I[Sum all values] - I --> J[Divide by count] - - style G fill:#ffd43b -``` - -## Implementation Checklist - -- [ ] Create gst/rollingsum directory -- [ ] Implement gstrollingsum.h -- [ ] Implement gstrollingsum.c -- [ ] Create CMakeLists.txt -- [ ] Update gst/CMakeLists.txt -- [ ] Build and test basic functionality -- [ ] Test with idsueyesrc -- [ ] Update README.md -- [ ] Create feature branch -- [ ] Commit and document - -## References - -- Original algorithm: `cli.py` lines 64-79 (column extraction and mean comparison) -- Template element: [`gst/select/gstselect.c`](gst/select/gstselect.c) -- GStreamer base transform: [GstBaseTransform documentation](https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html) \ No newline at end of file