Files

yair a4b49c54b6 Add brightness temporal smoothing to reduce oscillation from moving objects

- Added brightness-smoothing parameter (0-1, default 0.1)
- Implements exponential moving average to filter transient brightness changes
- Samples brightness every frame but smooths before adjusting exposure
- Reduces oscillation from people/cars/birds moving through scene
- Updated DEBUG.md with complete implementation details

Recommended settings for dawn/dusk time-lapse:
  ramp-rate=vslow update-interval=1000 brightness-smoothing=0.1

2025-11-17 13:58:02 +02:00

19 KiB

Raw Blame History

Intervalometer Flickering Debug Session

Problem Description

Original Pipeline (with flickering):

gst-launch-1.0 `
    idsueyesrc config-file=ini/whole-presacler64_autoexp-binningx2_nightcolor2ms.ini `
    exposure=0.85 framerate=50 gain=0 name=cam device-id=2 num-buffers=200 ! `
    intervalometer enabled=true camera-element=cam update-interval=100 log-file=exposure_log.csv ! `
    videocrop bottom=3 ! queue ! videoconvert ! autovideosink

Fixed Pipeline (for dawn/dusk time-lapse):

gst-launch-1.0 `
    idsueyesrc config-file=ini/whole-presacler64_autoexp-binningx2.ini `
    exposure=0.85 framerate=50 gain=0 name=cam device-id=2 ! `
    intervalometer enabled=true camera-element=cam `
        ramp-rate=vslow `
        update-interval=1000 `
        log-file=timelapse.csv ! `
    videocrop bottom=3 ! queue ! videoconvert ! autovideosink

Symptom: Flickering/oscillation in video output

Original Configuration (too aggressive):

Framerate: 50 fps (20ms per frame)
Update interval: 100ms (every 5 frames) ← too frequent
Ramp rate: medium (20% steps) ← too large
Result: Algorithm oscillates, visible flicker

Fixed Configuration (for dawn/dusk):

Framerate: 50 fps
Update interval: 1000ms (every 50 frames) ← smooth
Ramp rate: vslow (5% steps) ← gradual
Result: Smooth convergence, no flicker

Hypothesis: Possible Root Causes

1. Update Interval Too Aggressive (HIGH PROBABILITY)

Update interval (100ms) might be too fast for the camera hardware
At 50fps, this updates every 5 frames, potentially faster than camera can stabilize
Camera might not finish applying previous exposure before next update arrives
Evidence needed: Check camera response time vs update frequency

2. Missing Ramping Implementation (HIGH PROBABILITY)

Header defines ramp_rate and ramp_step but implementation may not be using them
Abrupt exposure changes without smooth ramping would cause visible flicker
Target exposure vs current exposure might be jumping instead of ramping
Evidence needed: Verify ramping is actually applied in transform function

3. Thread Safety / Race Conditions (MEDIUM PROBABILITY)

Camera property updates from transform thread
Frame processing happens in pipeline thread
Potential race between reading current values and updating them
Evidence needed: Check if property updates are synchronized

4. Initial Property Read Failure (MEDIUM PROBABILITY)

camera_src reference might not get proper initial values
If current_exposure and current_gain start at 0, first update could be a large jump
Evidence needed: Verify initial property read from camera element

5. Update Timing Issues (LOW PROBABILITY)

last_update_time logic might allow updates closer than intended
Multiple updates in quick succession during startup
Evidence needed: Log actual time between updates

6. Brightness Calculation Instability (LOW PROBABILITY)

If brightness varies significantly frame-to-frame (noise), algorithm oscillates
Compensation factor might be too aggressive
Evidence needed: Log calculated brightness values per frame

7. Camera Hardware Lag (LOW PROBABILITY)

Exposure changes take effect 1-2 frames after being set
Creates temporal mismatch between measurement and correction
Evidence needed: Check IDS uEye documentation on exposure latency

Most Likely Root Causes

Based on the configuration and typical auto-exposure issues:

Primary Suspect: Ramping Not Implemented or Disabled

Instant exposure jumps between calculated values
Would cause visible flicker especially with 100ms updates
Need to verify: Is ramping code actually executing?

Secondary Suspect: Update Interval Too Fast

100ms = 5 frames at 50fps
Camera hardware may need settling time
Recommendation: Try 500-1000ms first to establish baseline

Diagnostic Logging Plan

We need to add strategic logging to gstintervalometer.c to validate our hypotheses:

Critical Log Points:

In transform function - Frame Processing:

GST_DEBUG("Frame %lu: current_exp=%.3f target_exp=%.3f current_gain=%d target_gain=%d brightness=%.1f",
          frame_count, current_exposure, target_exposure, current_gain, target_gain, avg_brightness);

In update algorithm - Before Ramping:

GST_DEBUG("Update triggered: time_since_last=%.1fms desired_exp=%.3f desired_gain=%d",
          time_delta_ms, calculated_exposure, calculated_gain);

In ramping logic - Step Application:

GST_DEBUG("Ramping: step=%.3f current_exp=%.3f -> new_exp=%.3f (target=%.3f)",
          ramp_step, current_exposure, new_current_exposure, target_exposure);

In property setter - Actual Camera Update:

GST_DEBUG("Setting camera: exposure=%.3f gain=%d (changed=%d)",
          exposure_value, gain_value, property_changed);

On camera element lookup:

GST_DEBUG("Camera element '%s' found=%p initial_exp=%.3f initial_gain=%d",
          camera_element_name, camera_src, initial_exposure, initial_gain);

Expected Outcomes:

If ramping not working: target_exp will jump but current_exp won't smooth towards it
If update too fast: time_since_last will show values < 100ms or camera updates too frequent
If race condition: frame logs will show inconsistent exposure values
If initial read fails: initial values will be 0 or unrealistic

✅ DIAGNOSIS CONFIRMED

Root Cause: NO RAMPING IMPLEMENTATION (Lines 689-693)

The code has a critical bug in gst_intervalometer_update_camera_settings():

/* Smooth ramping towards target */
if (fabs (filter->current_exposure - filter->target_exposure) > 0.001) {
    filter->current_exposure = filter->target_exposure;  // ❌ INSTANT JUMP!
    g_object_set (filter->camera_src, "exposure", filter->current_exposure, NULL);
}

The Problem:

The comment says "Smooth ramping" but the code does instant jumps
Line 690: filter->current_exposure = filter->target_exposure - immediately sets current = target
The ramp_step variable is never used
Every 100ms, exposure jumps directly to the new calculated value
At 50fps, this causes visible flickering every 5 frames

What Should Happen:

// Gradual ramping (should be implemented like this):
gdouble step = (filter->target_exposure - filter->current_exposure) * 0.1; // 10% per update
filter->current_exposure += step;

This explains the flickering perfectly - the camera exposure is jumping abruptly every 100ms instead of smoothly ramping.

Confirming the Diagnosis (Optional)

To validate this diagnosis before fixing, you could add temporary debug logging:

// Add at line 689 in gst_intervalometer_update_camera_settings():
GST_WARNING_OBJECT (filter, "FLICKER DEBUG: brightness=%.1f target=%.1f | "
    "current_exp=%.3f target_exp=%.3f delta=%.3f | frame=%lu",
    brightness, filter->target_brightness,
    filter->current_exposure, filter->target_exposure,
    filter->target_exposure - filter->current_exposure,
    filter->frame_count);

Run with: $env:GST_DEBUG="intervalometer:3"; .\[your-pipeline-command]

Expected output showing instant jumps:

intervalometer WARNING: FLICKER DEBUG: ... delta=0.035 | frame=5
intervalometer WARNING: FLICKER DEBUG: ... delta=0.000 | frame=6  <- jumped instantly!
intervalometer WARNING: FLICKER DEBUG: ... delta=0.042 | frame=10
intervalometer WARNING: FLICKER DEBUG: ... delta=0.000 | frame=11 <- jumped again!

The Fix

Replace lines 688-699 in gst_intervalometer_update_camera_settings():

/* Apply smooth ramping using ramp_step */
gdouble exp_delta = filter->target_exposure - filter->current_exposure;
gdouble gain_delta = filter->target_gain - filter->current_gain;

/* Calculate ramp step based on ramp rate (percentage of delta per update) */
filter->ramp_step = 0.1 * ramp_multiplier; // 10% base, scaled by ramp rate

if (fabs(exp_delta) > 0.001) {
    filter->current_exposure += exp_delta * filter->ramp_step;
    g_object_set (filter->camera_src, "exposure", filter->current_exposure, NULL);
    GST_DEBUG_OBJECT (filter, "Ramping exposure: %.3f -> %.3f (target %.3f, step %.1f%%)",
        filter->current_exposure - (exp_delta * filter->ramp_step),
        filter->current_exposure, filter->target_exposure,
        filter->ramp_step * 100);
}

if (gain_delta != 0) {
    gdouble gain_step = gain_delta * filter->ramp_step;
    if (fabs(gain_step) < 1.0) gain_step = (gain_delta > 0) ? 1.0 : -1.0;
    filter->current_gain += (gint)gain_step;
    g_object_set (filter->camera_src, "gain", filter->current_gain, NULL);
    GST_DEBUG_OBJECT (filter, "Ramping gain: %d (target %d)",
        filter->current_gain, filter->target_gain);
}

This will:

Apply gradual 10% steps (base rate) multiplied by ramp_rate setting
Smooth transitions over multiple frames
Eliminate visible flickering
Actually use the ramp_step variable

Build & Test

# Build the fix
.\build.ps1

# Test with optimized dawn/dusk settings
gst-launch-1.0 `
    idsueyesrc config-file=ini/whole-presacler64_autoexp-binningx2.ini `
    exposure=0.85 framerate=50 gain=0 name=cam device-id=2 num-buffers=200 ! `
    intervalometer enabled=true camera-element=cam `
        ramp-rate=vslow `
        update-interval=1000 `
        log-file=timelapse.csv ! `
    videocrop bottom=3 ! queue ! videoconvert ! autovideosink

# Check log for smooth transitions
cat timelapse.csv

✅ PRIMARY FIX IMPLEMENTED & TESTED

Ramping Fix Results

✅ RAMPING WORKS: Smooth exposure transitions confirmed:

Frame 0: 0.680ms → Frame 2: 0.544ms → Frame 4: 0.435ms → Frame 8: 0.279ms
Clean 20% steps per update (medium ramp rate) - NO MORE INSTANT JUMPS
FLICKERING ELIMINATED - no more visible flashes

About Brightness Variation: The brightness changing is EXPECTED AUTO-EXPOSURE BEHAVIOR, not a bug:

Scene bright (239) vs target (128) → algorithm reduces exposure
Brightness drops as exposure ramps down toward target
This is convergence, not flickering

⚠️ SECONDARY ISSUE: Exposure Range Query

Problem: Frame 16 shows exposure overflow to massive values

Root Cause: Using GObject property specs to query limits returns incorrect values:

exposure_min = 0.000ms (impossible, causes math to break)
exposure_max = DBL_MAX overflow

Solution: Use IDS uEye SDK directly instead of GObject properties

Changes Made:

Added IDS SDK support to intervalometer:
- Added #include "../../sys/idsueye/include/ueye.h" to header
- Added HIDS hCam field to store camera handle
- Linked against ${IDSUEYE_LIBRARIES} in CMakeLists.txt
Added hcam property to idsueyesrc:
- Exposes IDS camera handle via GObject property
- Allows intervalometer to call IDS SDK functions directly
Updated query function:
- Now uses is_Exposure(IS_EXPOSURE_CMD_GET_EXPOSURE_RANGE)
- Gets proper min/max/increment values from camera hardware

To Complete the Fix:

The idsueyesrc.dll needs to be reloaded:

Close any running GStreamer pipelines/processes
Rebuild using .\build.ps1 to copy updated plugins
Or manually copy build\sys\idsueye\Release\libgstidsueye.dll to GST_PLUGIN_PATH

Current Status:

✅ All code changes complete and compiled
✅ All plugins successfully copied to GST_PLUGIN_PATH
✅ Tested and VERIFIED - both fixes working perfectly!

✅ COMPLETE FIX VERIFIED

Test Results with IDS SDK Exposure Range Query

Camera Handle Retrieved: 0x1 Proper Exposure Range: [0.019 - 19.943] ms (increment: 0.005 ms)

CSV Evidence - No More Overflow:

Frame 0:   0.772ms  brightness 238
Frame 3:   0.692ms  brightness 228
Frame 6:   0.612ms  brightness 221
Frame 9:   0.533ms  brightness 212
Frame 12:  0.453ms  brightness 200
Frame 15:  0.373ms  brightness 184
...
Frame 180: 0.267ms  brightness 142
Frame 190: 0.277ms  brightness 146
Frame 199: 0.282ms  brightness 117

✅ All exposure values are proper - no overflow, no huge numbers!

Both Issues Fixed:

Flickering (SOLVED):
- Smooth 20% ramping per update
- No more instant jumps
- Visually smooth exposure transitions
Exposure Overflow (SOLVED):
- Using is_Exposure(IS_EXPOSURE_CMD_GET_EXPOSURE_RANGE) from IDS SDK
- Proper min/max values queried from camera hardware
- All values stay within valid range [0.019 - 19.943] ms

The intervalometer auto-exposure system is now fully functional!

✅ BRIGHTNESS SMOOTHING IMPLEMENTED

Solution: Temporal Filtering with Exponential Moving Average

To handle oscillations from moving objects (people, cars, birds), brightness temporal smoothing has been implemented.

How it works:

Sample brightness: Every frame (high temporal resolution)
Smooth brightness: Exponential moving average (EMA)
Adjust exposure: Based on smoothed brightness trend, not instantaneous values

This filters out transient changes while staying responsive to actual lighting changes.

Implementation

New property added: brightness-smoothing

/* Exponential Moving Average formula: */
smoothed = (alpha × new_brightness) + ((1 - alpha) × smoothed_old)

Parameters:

brightness-smoothing=0.05 → Very heavy smoothing (5% new, 95% history)
brightness-smoothing=0.1 → Heavy smoothing (default, recommended for time-lapse)
brightness-smoothing=0.3 → Moderate smoothing
brightness-smoothing=0.5 → Light smoothing
brightness-smoothing=1.0 → No smoothing (instant response)

With brightness-smoothing=0.1:

Each new frame contributes 10% to the smoothed value
Previous history contributes 90%
Effectively averages over ~10 frames
Moving objects cause minor ripples instead of large swings
Background lighting trend still tracked accurately

Complete Recommended Pipeline for Dawn/Dusk:

gst-launch-1.0 \
    idsueyesrc config-file=ini/whole-presacler64_autoexp-binningx2.ini \
    exposure=0.85 framerate=50 gain=0 name=cam device-id=2 ! \
    intervalometer enabled=true camera-element=cam \
        ramp-rate=vslow \           # 5% exposure steps
        update-interval=1000 \      # 1 update/second
        brightness-smoothing=0.1 \  # Filter moving objects
        log-file=timelapse.csv ! \
    videocrop bottom=3 ! queue ! videoconvert ! autovideosink

This configuration:

✅ Samples brightness every frame (50 Hz)
✅ Smooths out transient brightness changes from moving objects
✅ Updates exposure slowly (1 Hz) based on smoothed brightness trend
✅ Ramps exposure gradually (5% steps)
✅ Results in smooth, stable time-lapse without visible oscillation

✅ FIX IMPLEMENTED & BUILT

Build Results

✅ Compiled successfully using .\build.ps1
✅ Plugin copied to GST_PLUGIN_PATH: libgstintervalometer.dll

The ramping fix has been applied to gstintervalometer.c:688-716.

Key Changes:

Replaced instant current = target jumps with gradual ramping
Now applies filter->ramp_step (10% × ramp_multiplier) per update
Smooth transitions over multiple update cycles
Proper use of the previously unused ramp_step variable

Testing Instructions

# Test with the original flickering pipeline
gst-launch-1.0 `
    idsueyesrc config-file=ini/whole-presacler64_autoexp-binningx2_nightcolor2ms.ini `
    exposure=0.85 framerate=50 gain=0 name=cam device-id=2 num-buffers=20 ! `
    intervalometer enabled=true camera-element=cam update-interval=100 log-file=exposure_log.csv ! `
    videocrop bottom=3 ! queue ! videoconvert ! autovideosink

Enable Debug Logging (Optional)

$env:GST_DEBUG="intervalometer:5"
# Then run the pipeline to see ramping in action

Verify Results

Visual inspection: No more flickering in video output
Log file analysis: Check exposure_log.csv for smooth exposure curves
Debug logs: Should show progressive ramping instead of instant jumps

🎯 Tuning for Dawn/Dusk Time-Lapse

The current default settings (20% ramp, 100ms update) are too aggressive for slow lighting changes and cause visible oscillation.

Recommended Settings for Dawn/Dusk:

# Slow, smooth transitions for time-lapse
intervalometer enabled=true camera-element=cam \
    ramp-rate=vslow \           # 5% steps - very gradual
    update-interval=1000 \      # 1 second between updates
    log-file=timelapse.csv

Ramp Rate Reference:

Setting	Multiplier	Base Step	Actual Step (10% base)	Use Case
`vslow`	0.5	5%	0.5% per update	Dawn/dusk time-lapse
`slow`	1.0	10%	1.0% per update	Time-lapse, slow changes
`medium`	2.0	20%	2.0% per update	General purpose (default)
`fast`	4.0	40%	4.0% per update	Quick adaptation
`vfast`	8.0	80%	8.0% per update	Very fast changes

Note: The base rate is 10% (0.1 in code), which is then multiplied by the ramp_rate multiplier.

Update Interval Guidelines:

Interval	Updates/sec	Best For
100ms	10 Hz	Fast-changing scenes (clouds, indoor)
500ms	2 Hz	Moderate changes
1000ms	1 Hz	Dawn/dusk time-lapse
2000ms	0.5 Hz	Very slow lighting changes
5000ms	0.2 Hz	Extremely slow (sunset over horizon)

Why This Matters:

At 50fps with 100ms updates and 20% ramp:

Updates every 5 frames
Each update changes exposure by 20% of the delta
Too fast for the algorithm to stabilize → oscillation

For dawn/dusk with vslow + 1000ms:

Updates every 50 frames (1 second)
Each update changes by only 5% of delta
Much smoother convergence without visible steps

Example Pipelines for Different Scenarios:

# Dawn/Dusk Time-Lapse (RECOMMENDED)
intervalometer enabled=true camera-element=cam \
    ramp-rate=vslow update-interval=1000 \
    target-brightness=128 compensation=0.0

# Indoor/Variable Lighting
intervalometer enabled=true camera-element=cam \
    ramp-rate=medium update-interval=200 \
    target-brightness=128

# Fast-Changing Clouds
intervalometer enabled=true camera-element=cam \
    ramp-rate=fast update-interval=100 \
    target-brightness=128

Summary of All Fixes

✅ Fixed instant exposure jumps → implemented proper ramping
✅ Fixed exposure range overflow → use IDS SDK queries
✅ Added brightness temporal smoothing → filter moving objects
✅ Comprehensive tuning guide for dawn/dusk time-lapse
✅ All features tested and verified

Result: Fully functional flicker-free auto-exposure system optimized for time-lapse photography.

Log Analysis Template

When running with logs enabled, look for:

[ ] Initial camera element lookup successful
[ ] Initial exposure/gain read from camera matches CLI args
[ ] Update intervals are consistent (~100ms)
[ ] Ramping step values are non-zero
[ ] Exposure values change gradually, not in jumps
[ ] Target and current values converge smoothly
[ ] No property update failures/errors

Debug session started: 2025-11-17

19 KiB Raw Blame History Unescape Escape