gst-plugin-linescan/network_guide.md

How to Send a Single Line (2456x1)

Real Data - Single Line Transmission

The camera captures 2456x4 pixels, but we extract and transmit only one line (2456x1) using videocrop.

Daytime Configuration (200fps)

gst-launch-1.0 idsueyesrc config-file=ini/100fps-10exp-2456x4pix-500top-cw-extragain.ini exposure=0.05 framerate=200 `
    ! videocrop bottom=3 `
    ! queue `
    ! udpsink host=127.0.0.1 port=5000

Nighttime Configuration (100fps, extra gain)

gst-launch-1.0 idsueyesrc config-file=ini/100fps-10exp-2456x4pix-500top-cw-extragain.ini exposure=10 framerate=100 `
    ! videocrop bottom=3 `
    ! queue `
    ! udpsink host=127.0.0.1 port=5000

Key Parameters:

• videocrop bottom=3 - Extracts only the top line (removes bottom 3 rows from 2456x4 image)
• Input: 2456x4 BGR from camera
• Output: 2456x1 BGR line transmitted via UDP
• Frame size: 7368 bytes (2456 × 1 × 3 channels)

Alternative: To extract the bottom line instead, use videocrop top=3

---

Python-Based Control with Dynamic Parameter Adjustment

For more advanced control including runtime exposure and framerate adjustments, use the Python-based launcher with UDP control interface.

Launch Camera with Control Interface

See scripts/launch-ids.py for the complete implementation.

# Setup GStreamer environment and run
. .\scripts\setup_gstreamer_env.ps1
uv run .\scripts\launch-ids.py

Features:

• Video streaming on UDP port 5000 (127.0.0.1)
• Control interface on UDP port 5001 (0.0.0.0)
• Dynamic exposure control (0.001-1.0 seconds)
• Dynamic framerate control (1-500 fps)
• Real-time parameter updates without restarting

Control Commands:

Command	Description	Example
SET_EXPOSURE <value>	Set exposure in seconds	SET_EXPOSURE 0.016
GET_EXPOSURE	Get current exposure value	GET_EXPOSURE
SET_FRAMERATE <value>	Set framerate in Hz	SET_FRAMERATE 30
GET_FRAMERATE	Get current framerate	GET_FRAMERATE
STATUS	Get pipeline status and settings	STATUS

Example Usage:

# Using netcat (nc) or PowerShell
echo "SET_EXPOSURE 0.010" | nc -u 127.0.0.1 5001
echo "GET_EXPOSURE" | nc -u 127.0.0.1 5001
echo "STATUS" | nc -u 127.0.0.1 5001

Testing the Control Interface

See scripts/test_exposure_control.py for automated testing.

# Run comprehensive test suite (15 tests)
uv run .\scripts\test_exposure_control.py

Test Coverage:

• Get/Set exposure and framerate
• Verify parameter changes take effect
• Test input validation and error handling
• Verify range limits (0.001-1.0s for exposure, 1-500 fps)
• Test invalid commands and syntax

Example Test Output:

Test 1: Get current exposure
  Command:  GET_EXPOSURE
  Response: OK 0.016
  ✓ PASS

Test 2: Set exposure to 10ms
  Command:  SET_EXPOSURE 0.010
  Response: OK 0.01
  ✓ PASS

---

UDP Control Protocol Specification

Complete technical details for the UDP-based control interface.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     launch-ids.py Process                    │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────────────┐         ┌──────────────────────────┐  │
│  │   Main Thread    │         │  Control Server Thread   │  │
│  │                  │         │                          │  │
│  │  GStreamer       │◄────────┤  UDP Socket (Port 5001)  │  │
│  │  Pipeline        │ Thread- │  Command Parser          │  │
│  │  - idsueyesrc    │  Safe   │  Property Setter         │  │
│  │  - videocrop     │ Updates │  Response Handler        │  │
│  │  - queue         │         │                          │  │
│  │  - udpsink:5000  │         └──────────────────────────┘  │
│  └──────────────────┘                    ▲                   │
│                                           │                   │
└───────────────────────────────────────────┼───────────────────┘
                                            │
                                            │ UDP Commands
                                            │
                                   ┌────────┴────────┐
                                   │  Control Client │
                                   │  (Any UDP tool) │
                                   └─────────────────┘

Connection Details

• Control Port: 5001 (UDP)
• Bind Address: 0.0.0.0 (accepts from any interface)
• Video Port: 5000 (UDP) - existing video stream, unchanged
• Protocol: UDP (connectionless, stateless)
• Encoding: ASCII text
• Delimiter: Newline (\n)

Command Format

Commands follow this general structure:

COMMAND [PARAMETERS]\n

Commands are case-insensitive, but UPPERCASE is recommended for clarity.

Detailed Command Reference

1. SET_EXPOSURE

Description: Sets the camera exposure time.

Syntax:

SET_EXPOSURE <value>

Parameters:

• <value>: Exposure time in seconds (float)
  • Range: 0.001 to 1.0 seconds (1ms to 1000ms)
  • Examples: 0.016 (16ms), 0.001 (1ms), 0.100 (100ms)

Response:

OK <actual_value>

or

ERROR <error_message>

Examples:

Client: SET_EXPOSURE 0.016\n
Server: OK 0.016\n

Client: SET_EXPOSURE 2.0\n
Server: ERROR OUT_OF_RANGE: Exposure must be 0.001-1.0 seconds\n

2. GET_EXPOSURE

Description: Retrieves the current exposure time.

Syntax:

GET_EXPOSURE

Parameters: None

Response:

OK <current_value>

Example:

Client: GET_EXPOSURE\n
Server: OK 0.016\n

3. SET_FRAMERATE

Description: Sets the camera frame rate.

Syntax:

SET_FRAMERATE <value>

Parameters:

• <value>: Frame rate in Hz (float)
  • Range: 1.0 to 500.0 fps
  • Examples: 22, 30.5, 100

Response:

OK <actual_value>

or

ERROR <error_message>

Example:

Client: SET_FRAMERATE 30\n
Server: OK 30.0\n

4. GET_FRAMERATE

Description: Retrieves the current frame rate.

Syntax:

GET_FRAMERATE

Parameters: None

Response:

OK <current_value>

Example:

Client: GET_FRAMERATE\n
Server: OK 22.0\n

5. STATUS

Description: Get overall pipeline status and current settings.

Syntax:

STATUS

Parameters: None

Response:

OK exposure=<value> framerate=<value> state=<PLAYING|PAUSED|NULL>

Example:

Client: STATUS\n
Server: OK exposure=0.016 framerate=22.0 state=PLAYING\n

Error Handling

Error Response Format:

ERROR <error_code>: <error_message>

Common Error Codes:

Code	Description	Example
INVALID_COMMAND	Unknown command	ERROR INVALID_COMMAND: Unknown command 'FOO'
INVALID_SYNTAX	Malformed command	ERROR INVALID_SYNTAX: Missing parameter
OUT_OF_RANGE	Value out of valid range	ERROR OUT_OF_RANGE: Exposure must be 0.001-1.0
PROCESSING	Internal processing error	ERROR PROCESSING: Failed to set property

Client Implementation Examples

Python Client

import socket

def send_command(command, host="127.0.0.1", port=5001):
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.sendto(command.encode() + b'\n', (host, port))
    sock.settimeout(1.0)
    response, _ = sock.recvfrom(1024)
    sock.close()
    return response.decode().strip()

# Set exposure to 10ms
print(send_command("SET_EXPOSURE 0.010"))

# Get current exposure
print(send_command("GET_EXPOSURE"))

# Set framerate to 30fps
print(send_command("SET_FRAMERATE 30"))

Command Line (netcat/nc)

# Set exposure
echo "SET_EXPOSURE 0.020" | nc -u 127.0.0.1 5001

# Get exposure
echo "GET_EXPOSURE" | nc -u 127.0.0.1 5001

# Get status
echo "STATUS" | nc -u 127.0.0.1 5001

PowerShell Client

$udpClient = New-Object System.Net.Sockets.UdpClient
$endpoint = New-Object System.Net.IPEndPoint([System.Net.IPAddress]::Parse("127.0.0.1"), 5001)

# Send command
$bytes = [System.Text.Encoding]::ASCII.GetBytes("SET_EXPOSURE 0.015`n")
$udpClient.Send($bytes, $bytes.Length, $endpoint)

# Receive response
$udpClient.Client.ReceiveTimeout = 1000
$receiveBytes = $udpClient.Receive([ref]$endpoint)
$response = [System.Text.Encoding]::ASCII.GetString($receiveBytes)
Write-Host $response

$udpClient.Close()

Implementation Notes

Thread Safety

• The control server runs in a separate daemon thread
• GStreamer properties are inherently thread-safe (GObject properties)
• The src.set_property() method can be safely called from the control thread

Non-Blocking Operation

• Control server uses non-blocking socket with timeout
• Does not interfere with GStreamer pipeline operation
• Minimal latency for command processing

Response Timing

• Responses are sent immediately after processing
• Property changes take effect on the next frame capture
• No guaranteed synchronization with video stream

Future Enhancements

Possible extensions to the protocol:

• Add SET_GAIN / GET_GAIN commands
• Add SAVE_CONFIG to save current settings to INI file
• Add RESET to restore default settings
• Support batch commands (multiple commands in one packet)
• Add authentication/security for production use

---

Python/OpenCV Receiver

# Basic rolling display
uv run .\scripts\recv_raw_rolling.py

# With display throttling and recording
uv run .\scripts\recv_raw_rolling.py --display-fps 60 --save-mjpeg .\results\output_60fps.avi

# Max performance (no display, stats only)
uv run .\scripts\recv_raw_rolling.py --no-display

See scripts/recv_raw_rolling.py for the Python implementation with debug options.

UDP Traffic Analysis & Debugging

To inspect and analyze the raw UDP packets being transmitted:

# Detailed payload analyzer - shows format, dimensions, pixel statistics
uv run .\scripts\udp_payload_analyzer.py

Example Output:

================================================================================
PACKET #1 @ 17:45:23.456
================================================================================
Source: 127.0.0.1:52341
Total Size: 7368 bytes

PROTOCOL ANALYSIS:
--------------------------------------------------------------------------------
  protocol            : RAW
  header_size         : 0
  payload_size        : 7368

VIDEO PAYLOAD ANALYSIS:
--------------------------------------------------------------------------------
  📹 Real camera data - Single line 2456x1 BGR
  Format: BGR
  Dimensions: 2456x1
  Channels: 3

PIXEL STATISTICS:
--------------------------------------------------------------------------------
  Channel 0 (B/R)     : min=  0, max=110, mean= 28.63, std= 16.16
  Channel 1 (G)       : min= 17, max=233, mean= 62.39, std= 36.93
  Channel 2 (R/B)     : min= 25, max=255, mean= 99.76, std= 49.81

HEX PREVIEW (first 32 bytes):
--------------------------------------------------------------------------------
  19 2e 4a 12 30 41 0a 2f 3f 01 32 3e 00 32 40 00 31 45 18 2d 4c 1e 2d...

SESSION SUMMARY:
Total Packets: 235
Total Bytes: 1,731,480 (7368 bytes/packet)

The analyzer automatically detects the format, shows pixel statistics per color channel, and provides a hex preview for debugging. Perfect for verifying data transmission and diagnosing issues.

# Simple packet receiver (no analysis, just basic info)
uv run .\scripts\udp_sniffer_raw.py

Real-time Channel Visualization

For live visualization of RGB/BGR channel values across the line sensor, use the real-time plotter.

See scripts/visualize_line_realtime.py for the implementation.

# Basic usage (assumes BGR format, 2456 width, port 5000)
uv run .\scripts\visualize_line_realtime.py

# With custom parameters
uv run .\scripts\visualize_line_realtime.py --format BGR --port 5000 --width 2456 --fps-limit 30

Features:

• Dual Plot Display:
  • Top plot: Grayscale luminance (weighted: BGR→0.114B+0.587G+0.299R)
  • Bottom plot: Individual RGB/BGR color channels
• Real-time Statistics:
  • Per-channel min/max/mean/std deviation
  • Display FPS counter
  • Frame counter
• Optimized Performance:
  • Packet draining (always displays latest frame)
  • Configurable display rate limiting
  • Minimal buffer size to reduce latency

Command-line Options:

Option	Default	Description
--format	BGR	Input format (BGR or RGB)
--port	5000	UDP port to receive on
--width	2456	Line width in pixels
--fps-limit	30	Maximum display update rate

Example Output:

The visualization shows live channel data with statistics overlay:

Frame: 1523  FPS: 29.8
Blue : min= 12 max=203 mean= 45.32 std= 28.45
Green: min= 28 max=241 mean= 68.91 std= 35.12
Red  : min= 35 max=255 mean=102.76 std= 48.23
Gray : min= 41.23 max=241.67 mean= 72.45 std= 36.89

Use Cases:

• Verify camera line sensor is working correctly
• Monitor real-time brightness and color balance
• Debug exposure and gain settings
• Analyze scene illumination changes
• Quality control and calibration

Close the plot window to exit.

---

Configuration Notes

INI file is configured with:

• Start Y = 500 (captures from row 500 of the sensor)
• Height = 4 pixels
• Width = 2456 pixels
• This optimizes for the center region of the sensor

Note: exposure=5 (5ms) may be too fast for some applications. Adjust based on your requirements.

---

Demo Data (Testing)

Sender (crop to first column, send raw over UDP)

gst-launch-1.0 -v `
    videotestsrc pattern=smpte ! `
    videocrop left=0 right=639 top=0 bottom=0 ! `
    video/x-raw,format=RGB,width=1,height=640,framerate=30/1 ! `
    udpsink host=127.0.0.1 port=5000

GStreamer Receiver (raw UDP → display)

gst-launch-1.0 -v `
    udpsrc port=5000 caps="video/x-raw,format=RGB,width=1,height=640,framerate=30/1" ! `
    videoconvert ! `
    autovideosink