Skip to content

Calibration Workflow

Detailed guide to using the calibrate tool to tune quality thresholds.

Purpose

The calibrate tool helps you:

  1. Understand metric distributions in your data
  2. Choose appropriate quality thresholds for different pass rates
  3. Diagnose quality issues (e.g., pervasive blur, darkness)
  4. Predict rejection rates before running full sampling

Use calibrate before every new dataset to ensure thresholds match your data characteristics.

Basic Usage

Single Video

./calibrate /path/to/sample_video.mp4

Choose a representative clip: - ✅ 5–15 minutes duration (1000–10000 examined frames at 1 fps) - ✅ Variety of conditions (lighting, subjects, motion) - ❌ Not an outlier (e.g., all blur or all darkness)

Multiple Videos

./calibrate video1.mp4 video2.mp4 video3.mp4

Metrics are pooled across all videos (reports combined distribution).

Use multiple videos when: - Dataset has high variability (different locations, times, conditions) - Single video is not representative - You want to ensure thresholds work across all data

Output Explained

Section 1: Metric Distributions

Metric distributions (all examined frames):
  brightness    min=8.2   p5=18.3   median=87.2   p95=201.4  max=243.1
  sharpness     min=3.1   p5=12.7   median=45.3   p95=187.9  max=321.5
  entropy       min=1.3   p5=2.2    median=3.9    p95=5.3    max=6.2
  motion        min=0.5   p5=1.8    median=8.4    p95=28.7   max=89.3

Columns:

  • min: Minimum value (worst frame)
  • p5: 5th percentile (~80% of frames are higher)
  • median: 50th percentile (half above, half below)
  • p95: 95th percentile (~80% of frames are lower)
  • max: Maximum value (best frame)

How to interpret:

Brightness

Pattern Interpretation
median < 50 Dark environment (deep water, night)
median > 150 Bright environment (shallow water, strong lighting)
p5 < 10 Some very dark frames (sensor issues or occlusion)
p95 > 240 Some overexposed frames (lighting artifacts)

Example: median=87.2 → moderate brightness (mid-water with artificial lighting)

Sharpness

Pattern Interpretation
median < 30 Generally blurry (motion blur, turbidity, poor optics)
median > 80 Generally sharp (good optics, stable platform)
p5 < 10 Some very blurry frames (rapid motion or defocus events)
Large range (max/p5 > 20) High variability (mix of sharp and blurry)

Example: median=45.3, p5=12.7 → mostly acceptable sharpness with some blurry frames

Entropy

Pattern Interpretation
median < 3.0 Low-texture environment (open ocean, uniform scenes)
median > 5.0 High-texture environment (reefs, particle clouds, benthos)
p5 < 2.0 Some featureless frames (blue water, occlusions)

Example: median=3.9 → moderate texture (mid-water with some features)

Motion

Pattern Interpretation
median < 5 Mostly static (slow vehicle, stable scene)
median > 20 High motion (fast vehicle, dynamic scene)
p95 > 50 Some rapid motion events (scene changes, fast pans)

Example: median=8.4 → moderate motion (typical AUV transect)

Section 2: Per-Gate Rejection

Per-gate rejection counts:
  min-brightness (≥12):    125 / 1200 rejected  (10.4%)
  max-brightness (≤243):   43 / 1200 rejected   (3.6%)
  min-sharpness (≥15):     187 / 1200 rejected  (15.6%)
  min-entropy (≥2.5):      223 / 1200 rejected  (18.6%)

Combined quality filter:   412 / 1200 rejected  (34.3%)
                          788 / 1200 passed     (65.7%)

Interpretation:

Individual Gates

Shows which gate is most restrictive:

  • min-brightness (10.4%): Few dark frames (good lighting)
  • max-brightness (3.6%): Few overexposed frames (good exposure control)
  • min-sharpness (15.6%): Some blur (typical)
  • min-entropy (18.6%): Some featureless frames (blue water common)

Diagnosis:

Pattern Issue Solution
One gate > 50% That metric is problematic Relax that threshold or fix upstream issue
All gates < 5% Thresholds too permissive Tighten thresholds
Balanced (10–30% each) Good tuning Proceed with these thresholds

Combined Filter

Shows overall pass rate:

Pass Rate Interpretation Action
< 20% Too strict (rejecting too much) Relax thresholds
20–40% Strict (selective, high quality) Good for small datasets
40–70% Balanced (typical) Recommended
70–90% Permissive (inclusive) Good for large datasets or sparse features
> 90% Too permissive (barely filtering) Tighten thresholds

Example: 65.7% passed → balanced filtering (typical for underwater data)

Section 3: Threshold Recommendations

Threshold recommendations (for target pass rates):

  80% pass:  --min-brightness 18  --max-brightness 220  --min-sharpness 12  --min-entropy 2.2
  60% pass:  --min-brightness 28  --max-brightness 205  --min-sharpness 25  --min-entropy 2.8
  40% pass:  --min-brightness 42  --max-brightness 185  --min-sharpness 48  --min-entropy 3.5
  20% pass:  --min-brightness 68  --max-brightness 160  --min-sharpness 92  --min-entropy 4.3

How to use:

  1. Choose target pass rate:
  2. 80%: Permissive (keep most frames, large dataset)
  3. 60%: Balanced (recommended)
  4. 40%: Strict (high quality, small dataset)

  5. 20%: Very strict (maximum quality)

  6. Copy corresponding row to sample command: 

    ./sample \
  --min-brightness 28 \
  --max-brightness 205 \
  --min-sharpness 25 \
  --min-entropy 2.8 \
  ...

  7. Adjust individual thresholds if needed (see Fine-Tuning)

Workflow Examples

Example 1: First-Time Calibration

Goal: Calibrate for a new deep-sea benthic survey.

# Step 1: Run calibrate on a representative clip
./calibrate /data/Triton43_benthic_sample.mp4

# Output shows:
#   brightness:  min=12  median=45  max=180
#   sharpness:   min=8   median=32  max=150
#   entropy:     min=1.8 median=3.5 max=5.8
#   Combined pass rate (default thresholds): 48%

# Step 2: Interpretation
# - Low brightness (median=45): Dark environment
# - Moderate sharpness (median=32): Some blur (typical for benthos)
# - Moderate entropy (median=3.5): Good detail (organisms, geology)
# - Pass rate 48%: Good balance

# Step 3: Use 60% pass row for more inclusion
#   --min-brightness 20  --min-sharpness 18  --min-entropy 2.8

# Step 4: Run sample
./sample \
  --root-dir /data/benthic_survey \
  --camera 1 \
  --min-brightness 20 \
  --min-sharpness 18 \
  --min-entropy 2.8 \
  --output-dir ./frames

Example 2: Diagnosing Blur

Goal: Dataset has pervasive blur, need to determine acceptable threshold.

# Step 1: Run calibrate
./calibrate /data/blurry_video.mp4

# Output shows:
#   sharpness:  min=2  p5=5  median=18  p95=45  max=120
#   min-sharpness (≥15): 58% rejected

# Step 2: Interpretation
# - Median sharpness is low (18)
# - Current threshold (15) rejects 58% (too strict!)

# Step 3: Relax threshold to match p25 (accept ~75%)
#   Estimate p25 ≈ 10 (between p5=5 and median=18)

# Step 4: Test
./calibrate /data/blurry_video.mp4 --min-sharpness 10

# Output shows:
#   min-sharpness (≥10): 28% rejected (better!)

# Step 5: Use relaxed threshold in sample
./sample --min-sharpness 10 ...

Example 3: Multi-Environment Calibration

Goal: Dataset spans shallow reef, mid-water, and deep benthos. Need thresholds that work for all.

# Step 1: Calibrate on each environment separately
./calibrate /data/shallow_reef.mp4 > shallow_cal.txt
./calibrate /data/mid_water.mp4 > midwater_cal.txt
./calibrate /data/deep_benthos.mp4 > benthos_cal.txt

# Step 2: Compare distributions
# Shallow:   brightness median=140, sharpness median=80
# Mid-water: brightness median=85,  sharpness median=45
# Benthos:   brightness median=40,  sharpness median=30

# Step 3: Choose conservative thresholds (pass all environments)
# Use minimum median across environments:
#   --min-brightness 30  (< 40)
#   --min-sharpness 20   (< 30)

# Step 4: Validate with combined calibration
./calibrate /data/shallow_reef.mp4 /data/mid_water.mp4 /data/deep_benthos.mp4 \
  --min-brightness 30 --min-sharpness 20

# Output shows combined pass rate: 62% (good!)

# Step 5: Run sample
./sample \
  --root-dir /data \
  --min-brightness 30 \
  --min-sharpness 20 \
  ...

Fine-Tuning

Adjusting Individual Thresholds

Recommendation rows are starting points. Adjust based on:

  1. Domain knowledge: You know what's acceptable for your application
  2. Visual inspection: Look at frames near threshold (see Visual Validation)
  3. Per-gate rejection: If one gate is dominating, relax it

Example: 60% pass row suggests --min-sharpness 25, but you know your optics are poor: 

# Relax sharpness, keep other thresholds
./sample \
  --min-brightness 28 \
  --min-sharpness 18 \  # Relaxed from 25
  --min-entropy 2.8 \
  ...

Visual Validation

To see frames near threshold:

  1. Run calibrate with test threshold: 

    ./calibrate video.mp4 --min-sharpness 25

  2. Extract frames manually near threshold (sharpness ≈ 25): 

    # Use OpenCV or ffmpeg to extract frames
# Inspect visually: Are they acceptably sharp?

  3. Adjust threshold based on visual quality

Rule of thumb: If frames at threshold look acceptable, use it. If obviously poor, raise threshold.

Advanced Options

Custom Sample Rate

Test with different sampling rates:

# Sparse sampling (faster, fewer examined frames)
./calibrate video.mp4 --sample-fps 0.5

# Dense sampling (slower, more examined frames)
./calibrate video.mp4 --sample-fps 2.0

Effect: Denser sampling may reveal short-duration issues (e.g., brief blur events).

Recommendation: Use same --sample-fps in calibrate and sample for consistency.

Batch Calibration

Calibrate many videos and aggregate results:

# Calibrate all videos in directory
find /data/videos -name "*.mp4" | while read video; do
  ./calibrate "$video" --sample-fps 1.0 > "$(basename $video .mp4)_cal.txt"
done

# Analyze results (e.g., with Python/R)
# Compute median of medians, min of mins, etc.

Calibration Best Practices

Do's

✅ Calibrate on representative data: Include variety of conditions
✅ Use multiple videos if dataset is diverse
✅ Re-calibrate for new datasets: Thresholds are data-dependent
✅ Iterate: Run calibrate → sample → inspect → adjust → repeat
✅ Document thresholds: Keep record of what works for each dataset type

Don'ts

❌ Don't use outlier videos: Single corrupt/unusual clip biases calibration
❌ Don't over-fit: Thresholds should work on unseen data from same domain
❌ Don't ignore per-gate rejection: If one gate is dominating, investigate why
❌ Don't trust defaults blindly: Default thresholds may not suit your data

Troubleshooting

Issue: Pass rate is 0% or near-0%

Cause: Thresholds are too strict for your data (or data is very poor quality).

Solution: 1. Check metric distributions (are they unusually low?) 2. Relax all thresholds to permissive values: 

./calibrate video.mp4 --min-brightness 5 --min-sharpness 5 --min-entropy 1.5
3. Gradually tighten until pass rate is 40–70%

Issue: Per-gate rejection is unbalanced (one gate > 80%)

Cause: That metric is problematic in your data.

Solution: 1. Check if issue is fixable upstream (e.g., improve lighting, reduce motion blur) 2. If unfixable, relax that threshold: 

# If min-sharpness is rejecting 80%, lower it
./calibrate video.mp4 --min-sharpness 8

Issue: Recommendations are unintuitive (e.g., min > max)

Cause: Corrupt output or very unusual data distribution.

Solution: 1. Re-run calibrate (check for errors) 2. Manually inspect video (is it valid?) 3. Try different video

Summary

Calibration workflow:

  1. ✅ Choose representative clip(s)
  2. ✅ Run ./calibrate video.mp4
  3. ✅ Interpret distributions and rejection rates
  4. ✅ Choose target pass rate (40–70% recommended)
  5. ✅ Copy threshold row to sample command
  6. ✅ Adjust individual thresholds if needed
  7. ✅ Run sample and visually inspect output
  8. ✅ Iterate if needed (fast with caching)

Key metrics: - Pass rate: 40–70% ideal - Per-gate rejection: Balanced (no single gate > 50%) - Metric distributions: Reasonable for your environment

Next Steps