Autonomous Agent Operations Guide for eo-processor

# Autonomous Agent Operations Guide for `eo-processor`

This document defines how autonomous engineering agents should operate within the `eo-processor` repository. It covers:
1. Core mission and constraints
2. Repository awareness
3. Workflow patterns
4. How to add functionality (Rust + Python)
5. Safety and quality gates
6. Pre-commit / pre-push checklist (MANDATORY)
7. Decision trees
8. Failure handling & rollback
9. Communication templates

---

## 1. Mission

Enhance and maintain a hybrid Rust + Python Earth Observation (EO) processing library that provides high-performance spectral index computations and related utilities.

Agents MUST:
- Preserve performance and safety guarantees
- Avoid introducing unnecessary complexity
- Maintain consistency across both Rust and Python layers
- Keep documentation and badges up to date

Agents MUST NOT:
- Commit broken builds
- Downgrade performance without justification
- Introduce insecure or `unsafe` Rust code without explicit rationale
- Push experimental code directly to `main`

---

## 2. Repository Awareness

Key technologies:
- Rust core implemented with PyO3 (native extension)
- Python packaging via `maturin`
- Tests: Python (`pytest`), optional Rust tests (`cargo test`)
- Tooling: `tox`, `uv`, `ruff`, `mypy`, `cargo fmt`, `cargo clippy`
- Coverage: Python coverage XML → `coverage-badge.svg` (generated by script)
- Documentation: `README.md`, `QUICKSTART.md`, function listings
- Security posture: memory-safe, no network/file I/O inside core functions

Important files/directories:
- `src/` Rust source
- `python/eo_processor/` Python module exports and type hints
- `tests/` Python test suite
- `scripts/generate_coverage_badge.py` Coverage badge generator
- `pyproject.toml` Project metadata and dependencies
- `tox.ini` Multi-env and coverage configuration
- `Makefile` Helper targets

---

## 3. Standard Workflow Patterns

### A. Add New Computational Function
1. Define Rust implementation in `src/lib.rs`
2. Expose via `#[pyfunction]` and register in `#[pymodule]`
3. Add Python import/export in `python/eo_processor/__init__.py`
4. Add stub in `python/eo_processor/__init__.pyi`
5. Add tests in `tests/test_<feature>.py`
6. Document usage in `README.md` (Function list + example)
7. Run verification checklist (see Section 6)

### B. Improve Performance
1. Benchmark existing implementation vs candidate approach (timing with large arrays)
2. Validate numerical correctness (same output within tolerance)
3. Avoid unsafe blocks unless strictly necessary
4. Add comparison note in PR description

### C. Refactor
1. Confirm test coverage touches refactored paths
2. Preserve public API signatures
3. If API changes: update docs, bump version appropriately (SemVer logic)

---

## 4. Adding Functionality (Rust + Python Integration)

### Rust Function Template

```rust
#[pyfunction]
fn my_index<'py>(
    py: Python<'py>,
    a: PyReadonlyArray2<f64>,
    b: PyReadonlyArray2<f64>,
) -> PyResult<&'py PyArray2<f64>> {
    let a_arr = a.as_array();
    let b_arr = b.as_array();
    assert_eq!(a_arr.shape(), b_arr.shape(), "Input shapes must match");
    // Compute (a - b) / (a + b + EPSILON)
    let result = a_arr - &b_arr;
    let denom = a_arr + &b_arr + 1e-10;
    let out = &result / &denom;
    Ok(out.into_pyarray(py))
}
```

### Register in Module

```rust
#[pymodule]
fn _core(_py: Python, m: &PyModule) -> PyResult<()> {
    // existing functions...
    m.add_function(wrap_pyfunction!(my_index, m)?)?;
    Ok(())
}
```

### Python Export

```python
from ._core import my_index

__all__ = [
    # previous exports...
    "my_index",
]
```

### Type Stub

```python
def my_index(a: NDArray[np.float64], b: NDArray[np.float64]) -> NDArray[np.float64]: ...
```

### Test

```python
def test_my_index():
    import numpy as np
    from eo_processor import my_index
    a = np.array([[0.6, 0.7], [0.8, 0.9]])
    b = np.array([[0.2, 0.3], [0.4, 0.5]])
    out = my_index(a, b)
    assert out.shape == a.shape
    assert np.isfinite(out).all()
```

### Documentation Update
- Add to “Available Functions” section
- Provide usage snippet
- Mention domain relevance if applicable

---

## 5. Quality and Safety Principles

- Deterministic builds and tests
- Avoid global state
- Defensive shape checks for mismatched arrays
- Prefer explicit over implicit conversions
- Keep public APIs stable
- Respect performance constraints (large arrays typical)
- Maintain accessible README examples (runnable)

---

## 6. Pre-Commit / Pre-Push Checklist (MANDATORY)

Before ANY commit:
1. Git status clean of unrelated changes
2. File encodings UTF-8
3. No secrets / tokens added
4. Run: `cargo fmt`
5. Run: `cargo clippy --all-targets -- -D warnings`
6. Run: `cargo test` (if Rust tests exist) OR `cargo check --lib`
7. Python formatting (if Python code touched):
   - `ruff check python/eo_processor` (fix if necessary: `ruff fix ...`)
   - Optional: `black` / `isort` if adopted (not yet mandated here)
8. `mypy python/eo_processor` (ensure no new type errors)
9. Run Python tests: `tox -e py312` (or full `tox`)
10. If coverage impacted:
    - `tox -e coverage`
    - Regenerate badge: `python scripts/generate_coverage_badge.py coverage.xml coverage-badge.svg`
    - Confirm SVG renders correctly (open file locally)
11. Update `README.md`, stubs, and Sphinx docs (autosummary lists, architecture or functions pages) if public API or performance model changed; rebuild locally with `make docs`
12. Increment version in `pyproject.toml` if semantic change:
    - Patch: bug fixes
    - Minor: new backward-compatible features
    - Major: breaking changes
13. Re-run `uv sync` if dependencies changed
14. Confirm no large binary artifacts staged (`git diff --cached`)
15. Commit message format (see Section 8)
16. Push only after all above pass

Failure to complete ANY step = BLOCKED commit.

---

## 7. Decision Trees

### A. Should I Bump Version?
- Added function (backward compatible): Minor
- Fixed bug without API change: Patch
- Changed function signature or behavior incompatibly: Major

### B. Add vs Extend Existing Function?
- If logic is a specialization (e.g., normalized difference variant): extend docs, not new function.
- If formula/domain distinct (new spectral index): new function.

### C. Optimize or Keep Simpler?
- If speedup > 1.2x on 1000x1000 arrays with same correctness → optimize.
- If complexity increases >2x lines without measurable benefit → reject.

---

## 8. Commit Message Conventions

Format:
```
<type>(scope): short summary

Body: Optional deeper explanation.
Refs: issues, benchmarks, design notes.

Types:
- feat: new functionality
- fix: bug fix
- perf: performance improvement
- docs: docs change only
- chore: maintenance / refactor without feature
- test: adding/improving tests
- build: tooling / packaging updates
- ci: workflow changes
```

Example:
```
feat(indices): add soil moisture index function

Implements SMI in Rust with 1D/2D variants. Benchmarked 1.35x faster vs NumPy baseline.
Ref: #42
```

---

## 9. Failure Handling & Rollback

If a step fails:
- DO NOT force commit.
- Investigate root cause (e.g., failing test, clippy warning).
- Add temporary debug logs ONLY in local branch; remove before commit.
- If performance regression found: revert change or open performance issue with benchmarks.

Rollback Procedure:
1. `git checkout main`
2. `git pull origin main`
3. `git branch -D <broken-branch>` (only if safe)
4. Recreate branch from fresh main

---

## 10. Coverage Badge Regeneration Rules

Regenerate when:
- Python logic changed
- Added/removed tests
- Coverage threshold changed in `tox.ini`
Steps:
1. Run `tox -e coverage`
2. Confirm `coverage.xml` exists
3. Run badge script:
   ```
   python scripts/generate_coverage_badge.py coverage.xml coverage-badge.svg
   ```
4. Open badge file locally (ensure text aligned, no clipping)
5. Stage updated `coverage-badge.svg`

---

## 11. Performance Benchmark Guidance

Minimal benchmarking template:
```python
import numpy as np, time
from eo_processor import ndvi

a = np.random.rand(5000, 5000)
b = np.random.rand(5000, 5000)

t0 = time.time()
r1 = ndvi(a, b)
t1 = time.time() - t0

t0 = time.time()
r2 = (a - b) / (a + b)
t2 = time.time() - t0

print("Rust:", t1, "NumPy:", t2, "Speedup:", t2 / t1)
```

Acceptable performance claim: ≥1.2x speedup with same numerical result to 1e-9 tolerance.

---

## 12. Security Considerations

Agents MUST:
- Avoid adding network calls or file I/O to core library
- Avoid dynamic code execution (`eval`, `exec`)
- Preserve numerical safety (avoid silent NaNs)
- Use local constants (e.g., EPSILON) rather than magic numbers sprinkled

---

## 13. Documentation Consistency Rules

When adding features or modifying behavior, keep ALL documentation layers synchronized:

Mandatory updates:
- README: function table, usage snippet, and (for indices) mathematical formula
- QUICKSTART: add only if it materially improves newcomer onboarding (avoid bloat)
- Sphinx docs (docs/):
  - `docs/source/api/functions.rst` autosummary list (add/remove function names)
  - `docs/source/api/architecture.rst` if parallel strategy, performance thresholds, or internal design rationale changes
  - Regenerate local HTML via `make docs` and verify new/changed pages render
- Python wrapper docstring (in `python/eo_processor/__init__.py`)
- Type stub (`__init__.pyi`) signature coherence with implementation
- Inline Rust doc comments (`///`) including formula, assumptions (units, value ranges), and numeric stability notes (EPSILON usage, NaN handling)
- CHANGELOG entry summarizing addition or change (if project policy requires)

If adding a new spectral or temporal index:
- Provide canonical formula with symbols explained
- Reference standard source (paper / NASA spec) if widely recognized
- Note expected input scaling (e.g., surface reflectance 0–1 vs radiance)
- Describe valid output range and typical interpretation bands

If performance refactor:
- Update architecture page with:
  - What changed (e.g. fused loops, new parallel threshold)
  - Rationale (cache locality, reduced temporaries, etc.)
  - Benchmark summary (array shape, old/new timings, speedup)
- Remove or adjust outdated statements about the old approach

Version / metadata alignment:
- Ensure `__version__` in Python package, `pyproject.toml`, and any version badge references match
- If a new function is added (minor bump) or a breaking doc-codified behavior changes (major bump), update version before publishing docs

Read the Docs considerations:
- If new doc build dependencies introduced, update `[project.optional-dependencies].docs` and (if required) `docs/requirements.txt`
- If environment changes (e.g., need Rust for signature introspection), adjust `.readthedocs.yaml` accordingly

Validation checklist before commit (docs-related):
1. Search for stale references to removed/renamed functions
2. Open generated HTML (`docs/build/html/index.html`) and navigate to:
   - API functions page
   - Architecture page (if modified)
   - Any new function page
3. Confirm autosummary generated a page for each new function
4. Confirm internal cross-links (e.g., :doc:, :ref:) resolve (no broken links in Sphinx build output)
5. Confirm README and Sphinx descriptions do not diverge materially

Prohibited:
- Adding large narrative sections unrelated to EO processing scope
- Leaving placeholder TODOs in published docs
- Claiming unverified performance improvements

---

## 14. Checklist (Compact Form)

| # | Item | Status |
|---|------|--------|
| 1 | cargo fmt | |
| 2 | cargo clippy (no warnings) | |
| 3 | cargo test / check | |
| 4 | ruff (no errors) | |
| 5 | mypy (no new errors) | |
| 6 | pytest (tox or single env) | |
| 7 | Coverage (if needed) | |
| 8 | Badge regenerated (if coverage changed) | |
| 9 | README / stubs updated | |
| 10 | Version bump (if required) | |
| 11 | No secrets / large binaries | |
| 12 | Commit message per convention | |
| 13 | Benchmarks (if perf change) | |
| 14 | Docs match public API | |

Agents should maintain a local markdown copy and tick boxes before committing.

---

## 15. Escalation Criteria

Open an issue (instead of direct fix) when:
- Requires architectural change (new module structure)
- Introduces dependency with non-trivial maintenance cost
- Needs unsafe Rust for performance
- Drops coverage below threshold in `tox.ini`

---

## 16. Communication Templates

Issue Template (internal):
```
Title: <short summary>

Context:
<why needed>

Proposal:
<changes>

Impact:
<performance / API / maintenance>

Risks:
<list>

Next Steps:
- [ ] Approve design
- [ ] Implement
- [ ] Test
```

PR Summary:
```
## Summary
<overview>

## Changes
- <list>

## Performance
<benchmark results>

## Tests
<added/updated test files>

## Docs
Updated: README, stubs

## Checklist
- [x] fmt
- [x] clippy
- [x] tests pass
- [x] coverage badge
```

---

## 17. Agent Behavior Guidelines

- Work incrementally (small, testable chunks)
- Never batch unrelated changes
- Prefer explicit naming (e.g., `normalized_difference_2d`)
- Keep function granularity consistent (1D vs 2D vs generic dispatcher)
- Avoid dead code (remove unused experimental sections)
- Treat warnings as errors

---

## 18. Dealing With Badge Layout Issues

If SVG renders incorrectly:
1. Validate file diff (no truncation)
2. Open standalone in browser
3. Re-run generation using existing script
4. Ensure `coverage.xml` is valid (not empty)
5. If persistent: log issue + attach SVG + coverage.xml

---

## 19. Future Extension Ideas (For Agents to Propose)

- Add more EO indices (EVI, SAVI, NBR)
- Multi-band compositing utilities
- Optional GPU path (evaluate feasibility)
- Batch processing API for stacked arrays
- Dask higher-level wrappers (automatic chunk optimization)
- Rust-based windowed operations (moving statistics)

---

### Final Reminder

DO NOT PUSH if any mandatory checklist item is incomplete. Quality > speed.

Agents must act as responsible maintainers ensuring stability, performance, and clarity.

---
End of autonomous operations guide.