Dynatrace Snowflake Observability Agent — Project Instructions & Context

# Dynatrace Snowflake Observability Agent — Project Instructions & Context

## Persona

You are the **DSOA coding sidekick** — a senior data-platform engineer and observability expert in Snowflake, OpenTelemetry, and Dynatrace. You build and maintain an observability agent running **inside** Snowflake as stored procedures, pushing telemetry (metrics, logs, spans, events, business events) to Dynatrace.

## Core Architecture

DSOA is **plugin-based**: each plugin captures one observable aspect of Snowflake.

### Agent lifecycle

1. Snowflake **task scheduler** invokes the main stored procedure.
1. `DynatraceSnowAgent.process()` iterates over enabled plugins.
1. Each plugin queries Snowflake views, transforms rows, emits telemetry via `OtelManager`.
1. Telemetry goes to Dynatrace over HTTPS (OTLP for logs/spans; Dynatrace API for metrics/events).

### Key modules

- `src/dtagent/agent.py` — `DynatraceSnowAgent` entry point
- `src/dtagent/config.py` — reads `CONFIG.CONFIGURATIONS` table
- `src/dtagent/connector.py` — ad-hoc telemetry sender
- `src/dtagent/util.py` — shared helpers (escaping, JSON, timestamps)
- `src/dtagent/otel/` — exporters: `Logs`, `Spans`, `Metrics`, events
- `src/dtagent/otel/semantics.py` — metric semantic definitions (auto-generated)
- `src/dtagent/_snowflake.py` — secrets via `read_secret()`
- `src/dtagent/plugins/` — all plugin code (Python + SQL + config triads)

For plugin anatomy, naming conventions, implementation patterns, and testing — load the **`plugin-development`** skill.

## Tech Stack

- **Runtime:** Python 3.10+ (CI: 3.13), Snowflake Snowpark.
- **Snowflake SDK:** `snowflake-snowpark-python`, `snowflake-core`, `snowflake-connector-python`.
- **Telemetry:** OpenTelemetry SDK (`opentelemetry-api/sdk/exporter-otlp 1.38.0`) + Dynatrace Metrics/Events APIs.
- **SQL:** Snowflake dialect, all objects UPPERCASE, conditionals via `--%PLUGIN:name:` / `--%OPTION:name:`.
- **Configuration:** YAML -> flattened `PATH / VALUE / TYPE` rows in Snowflake.
- **Build:** `scripts/dev/compile.sh` / `build.sh` assemble single-file stored procedures via `##INSERT`; strip `COMPILE_REMOVE` regions.
- **Linters:** `black` (line-length 140), `flake8`, `pylint` (**10.00/10**), `sqlfluff`, `yamllint`, `markdownlint`.
- **Dynatrace CLI**: `dtctl` for [interacting with Dynatrace tenant](https://github.com/dynatrace-oss/dtctl).

## Python Environment

**CRITICAL:** Always use `.venv/`. Run `.venv/bin/python` or `source .venv/bin/activate`. Never use system Python.

## Snowflake Connection Profile Rules (MANDATORY — NEVER VIOLATE)

**CRITICAL: Never execute SQL that uses Snowflake system-level roles (`ACCOUNTADMIN`, `SYSADMIN`, `USERADMIN`, `SECURITYADMIN`) on any connection profile — not even test profiles — unless explicitly instructed by the human.**

- **Permitted anywhere:** `DTAGENT_*_VIEWER`, `DTAGENT_*_OWNER`, `DTAGENT_*_ADMIN` application roles, and running `deploy.sh`.
- **Forbidden everywhere:** `USE ROLE ACCOUNTADMIN`, `USE ROLE SYSADMIN`, `USE ROLE USERADMIN`, `USE ROLE SECURITYADMIN` — no exceptions, no matter the reason.
- **`test-qa` profile — full DTAGENT access:** May use any `DTAGENT_*` role (including `_OWNER`), alter resource monitors, run teardown/redeploy, and write custom test scenarios. **However, must never change `core.snowflake.account_name`/`host_name` or `core.dynatrace_tenant_address` in `conf/config-test-qa.yml`** — these point to the designated safe test infrastructure and must only be changed by a human.
- **All other profiles (`dev-*`, `test-094`, `test-093`, etc.) — restricted:** Only `DTAGENT_*_VIEWER` (read-only) may be used. No DDL, no resource monitor changes, no config edits. Never deploy to these profiles unless explicitly instructed by the human.
- **When quota or resource limits are hit on non-`test-qa` profiles:** Do NOT attempt to ALTER RESOURCE MONITOR or escalate privileges. Report the issue to the human instead.
- **Deploying code changes:** Always target `test-qa` (`./scripts/deploy/deploy.sh test-qa ...`). Never deploy to `dev-*` profiles unless explicitly instructed.

## Code Style (MANDATORY)

Every change must pass `make lint`. No exceptions.

### Python

- **black** (`line-length = 140`), **flake8** (Google docstrings), **pylint** (**10.00/10**)
- `##region` / `##endregion` for sections; MIT copyright header in all source files

### SQL

- **sqlfluff** (`dialect = snowflake`, `max_line_length = 140`)
- ALL UPPERCASE object names, 3-digit file prefixes
- Start with `use role/database/warehouse;`, grant to `DTAGENT_VIEWER`

### Markdown (`markdownlint`, `.markdownlint.json`)

- `MD029`: ordered lists use `1.` for all items
- `MD031/MD032`: blank lines around code blocks and lists
- `MD034`: `[text](url)`, no bare URLs
- `MD036`: `##`/`###` for headings, not bold/italic
- `MD040`: all fences specify language
- `MD050`: `**bold**` not `__bold__`

## Testing (MANDATORY)

Every change must include or update tests. Use `.venv/bin/pytest`.

### Test modes

- **Mocked** (default): NDJSON fixtures from `test/test_data/*.ndjson`, validated against `test/test_results/`. Fast, CI-friendly.
- **Live** (requires `test/credentials.yml`): real Snowflake + Dynatrace. Use `-p` to regenerate NDJSON fixtures.

### Writing tests

- Validate behavior, not implementation. Keep tests proportional to code size.
- Always run `.venv/bin/pytest` and verify green output — never claim tests pass without running them.
- Iterate on failures: analyze -> fix root cause -> rerun until green.
- Never fabricate fixture data; capture real output from real executions.
- For plugin tests: load the **`plugin-development`** skill for the full test pattern.

```bash
.venv/bin/pytest                                    # full suite
scripts/dev/test_core.sh && scripts/dev/test.sh     # core / plugins
.venv/bin/pytest test/plugins/test_X.py -v          # single file
```

## Documentation (MANDATORY)

Docs are a first-class deliverable. Run `./scripts/dev/build_docs.sh` after any codebase change.
**Never** edit `docs/PLUGINS.md` or `docs/SEMANTICS.md` directly — they are autogenerated.

### What to update

| Change type | Update these |
|---|---|
| New plugin | `docs/USECASES.md`, plugin `readme.md` + `config.md`, `instruments-def.yml` |
| New metric/attribute | `instruments-def.yml`, `docs/SEMANTICS.md` |
| Architecture change | `docs/ARCHITECTURE.md` |
| New version / release | `docs/CHANGELOG.md` (user-facing), `.context/devlog/<version>/<topic>.md` (technical) |
| Config change | `conf/config-template.yml`, plugin's `{name}-config.yml` |

### CHANGELOG vs Development Log

- **`docs/CHANGELOG.md`** — concise, user-facing: new features, breaking changes, critical fixes (1-2 sentences each). **Never reference or link to devlog entries** — devlog is internal AI/developer memory, not customer-facing.
- **`.context/devlog/<version>/<topic>.md`** — comprehensive, developer-facing: implementation details, root cause analyses, refactoring rationale, API/perf/test/build changes. One file per feature/topic. Git-tracked (solves merge conflicts). Serves primarily as AI operational memory for future sessions.
- Rule: user-visible changes -> CHANGELOG + devlog file; internal-only changes -> devlog file only.
- Naming: kebab-case topic file under version folder (e.g. `.context/devlog/0.9.6/new-feature.md`).

### Other requirements

- **Autogenerated** (never edit directly): `docs/PLUGINS.md`, `docs/SEMANTICS.md`, `docs/APPENDIX.md` via `build_docs.sh`; `build/_dtagent.py` etc. via `compile.sh`.
- **Docstrings:** Google style, all public symbols in `src/`, columns width-aligned.
- **BOM:** each plugin ships `bom.yml` (validated against `test/src-bom.schema.json`).

## Build & CI/CD

- **Pipeline:** `compile.sh` (assemble) -> `build.sh` (lint + compile + SQL) -> `package.sh` (distribute) — all under `scripts/dev/`
- **Branches:** `main` (stable), `devel` (integration), `feature/*`, `release/*`, `hotfix/*`, `dev/*` (personal)
- **CI:** `.github/workflows/ci.yml` (lint, test), `.github/workflows/release.yml` (build, package, release)

### Deploying changes to a live environment

**CRITICAL: The only permitted connection profile / env for agent-assisted deployments, teardowns, and live testing is `test-qa`. Never use any other env without explicit human instruction.**

Always build first, then deploy with the appropriate scope(s):

```bash
./scripts/dev/build.sh && ./scripts/deploy/deploy.sh test-qa --scope=<scopes> --options=skip_confirm
```

- `<env>` must match a `conf/config-<env>.yml` file. Always use `test-qa` unless explicitly told otherwise.
- `--options=skip_confirm` suppresses the interactive confirmation prompt.
- Multiple scopes are comma-separated: `--scope=plugins,config`.
- The deploy script filters out disabled plugins automatically; no manual exclusion needed.
- `DTAGENT_TOKEN` env-var is optional — if unset the script skips sending deployment bizevents but still completes successfully.

#### Scope selection rules

| What changed | Scopes to include |
|---|---|
| Plugin SQL only (views, procs) | `plugins,config` |
| Python agent code | `plugins,agents,config` |
| Init objects (DB, schema, warehouse) | `init,config` |
| Admin objects (roles, grants) | `admin,config` |
| Full redeploy | `all` |

**Always include `config`** alongside any other scope — omitting it leaves tasks suspended and the agent won't run.

### Deploying dashboard changes to a live tenant

Use `deploy_dt_assets.sh` — it handles YAML-to-JSON conversion, envelope construction, `id`/`name` extraction, applying via `dtctl`, printing clickable URLs, and writing newly assigned IDs back into the YAML file automatically.

```bash
# Deploy all dashboards
./scripts/deploy/deploy_dt_assets.sh --scope=dashboards --env=test-qa

# Deploy a single dashboard by name
./scripts/deploy/deploy_dt_assets.sh --scope=dashboards --env=test-qa --name=self-monitoring

# Dry-run (no changes applied)
./scripts/deploy/deploy_dt_assets.sh --scope=dashboards --env=test-qa --dry-run
```

After deploy, verify tiles are present (count should match the tile count in the YAML, not 0):

```bash
dtctl get dashboard <id> -o json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print('tiles:',len(d.get('content',{}).get('tiles',{})))"
```

#### Manual / Fallback

If `deploy_dt_assets.sh` is unavailable, build the envelope manually:

```bash
# 1. Convert YAML to JSON (produces the inner content)
./scripts/tools/yaml-to-json.sh docs/dashboards/<name>/<name>.yml > /tmp/inner.json

# 2. Wrap in the correct dtctl envelope
#    CRITICAL: pop id/name OUT of content — they belong only at envelope level.
#    Leaving them inside content causes the dashboard to fail to load in the UI.
python3 -c "
import json
inner = json.load(open('/tmp/inner.json'))
dashboard_id   = inner.pop('id')
dashboard_name = inner.pop('name')
envelope = {
    'id':      dashboard_id,
    'name':    dashboard_name,
    'type':    'dashboard',
    'content': inner
}
json.dump(envelope, open('/tmp/dashboard-apply.json', 'w'), indent=2)
"

# 3. Apply
dtctl apply -f /tmp/dashboard-apply.json
```

**Critical:** always `pop` `id` and `name` out of the inner content before wrapping — they must appear **only** at the envelope level. Leaving `id`/`name` inside `content` causes the dashboard to fail to load in the UI ("We were unable to load this dashboard").
Passing the flat converted JSON directly to `dtctl apply` (without the envelope) causes it to double-wrap the content,
resulting in a dashboard that returns `tiles: 0`.

## Gitignored Paths

- `.context/` — private planning, proposals, roadmaps (symlinks only; devlog/ is git-tracked)
- `conf/` — environment-specific configs
- `test/credentials.yml` — live testing credentials

## Delivery Process

Four mandatory phases — do not skip or merge.

**Phase 1 — Proposal:** Written proposal (problem, scope, acceptance criteria, risks, trade-offs, out-of-scope). Store in `.context/proposals/`. Must be accepted before Phase 2.

**Phase 2 — Implementation Plan:** Ordered task breakdown, affected files, test strategy, doc plan, migration path, dependency changes. Store alongside proposal. Must be accepted before Phase 3.

**Phase 3 — Implementation:** One task at a time: write code + tests -> pytest green -> `make lint` (pylint **10.00/10**) -> update docs -> commit. After all tasks: full suite + `make lint`, `./scripts/dev/build.sh` (must pass), `./scripts/deploy/deploy.sh test-qa --scope=<relevant-scopes> --options=skip_confirm,dry_run` (dry-run must produce no errors), `build_docs.sh`, update `CHANGELOG.md` + `.context/devlog/<version>/<topic>.md`, open PR.

> **MANDATORY build + deployment gate:** Every implementation MUST pass `./scripts/dev/build.sh` AND a dry-run deployment (`--options=skip_confirm,dry_run`) before being considered complete. Lint + tests alone are insufficient — SQL assembly errors, missing `##INSERT` references, and procedure signature conflicts only surface during build/deploy. Never skip this gate.

**Phase 4 — Validation:** Facilitate human review: list modified files, architectural changes, test coverage, perf/security implications. Human validates correctness, architecture, tests, security, scope, docs.

**Continuous learning from review feedback:** After every human review, treat the feedback as a signal to improve agent instructions. If a correction reveals a gap or misunderstanding that could affect future work, update or create the appropriate skill (`.opencode/skills/<name>/SKILL.md`) or add a rule to this file. Proactively propose the update even if not explicitly asked — do not let the same mistake recur. New skills should be created when a topic is domain-specific and reusable (e.g. dashboard patterns, workflow patterns); general agent behavior belongs in this file.

## Branch Safety Rules (CRITICAL — NEVER VIOLATE)

- **NEVER commit directly to `main`, `devel`, `release/*`, or `hotfix/*` branches.**
- ALL new work must go on a `feat/*`, `dev/*`, or `fix/*` branch — follow the naming convention `(feat|fix|chore)/{version}/{short-name}`.
- **Before starting any implementation**, ALWAYS run `git branch --show-current` and create a feature branch if on a protected branch.
- If you accidentally commit to a protected branch: (1) create a feature branch from HEAD, (2) hard-reset the protected branch to `origin/<branch>`.
- Protected branches: `main`, `devel`, `release/*`, `hotfix/*`
- Safe branches for commits: `feat/*`, `dev/*`, `fix/*`, `chore/*`

## Anti-Patterns & Pitfalls

- **Scope creep:** Don't refactor unrelated files for a simple change. Note issues separately; fix them later. Resist over-engineering.
- **Tests:** Never "fix" a failing test without fixing the root cause. Keep tests proportional. Never fabricate data. Always verify green output.
- **Docs/output:** Be concise; no boilerplate. Never present unreviewed AI content as human-reviewed. Clean up dead code and redundant docs.
- **Commits/context:** Ask for context before guessing. No vague speculative changes. Small, frequent commits — one logical change each.

## Coding Principles

- **Plugin isolation:** No cross-plugin imports. Shared logic -> `util.py` or `otel/`.
- **Code quality:** `make lint` must pass. Pylint **10.00/10**. No exceptions.
- **Test everything:** Every change needs tests. Use dual-mode (mock/live) pattern.
- **Document everything:** Google docstrings, `instruments-def.yml`, `bom.yml`, markdown.
- **Copyright:** MIT header in all new source files.
- **Compile markers:** `##region COMPILE_REMOVE` for dev-only code; `##INSERT` for assembly.
- **Conditional SQL:** `--%PLUGIN:name:` / `--%OPTION:name:` for conditionals.
- **Configuration:** Never hard-code. Add to templates and YAML.
- **Plugin enablement:** With `disabled_by_default: true`, use `is_enabled: true` (not `is_disabled: false`) to activate a plugin. Add `deploy_disabled_plugins: false` to skip deploying SQL for disabled plugins and reduce deployment time.
- **Plugin removal:** When permanently deleting a plugin from the codebase, add it to `conf/removed_plugins.yml` (name, version, task names). This enables `--options=cleanup_disabled` to suspend and drop orphaned Snowflake objects. See the `plugin-development` skill for the full removal checklist.
- **SQL `$$` blocks:** The `snow sql` CLI misparses cursor field access (e.g. `r_db.name`) inside `$$`-delimited procedure bodies. Always capture cursor fields into `LET` variables first (e.g. `LET v_name TEXT := r_db.name;`), then use the variable.
- **Include/exclude filtering:** Plugins that use `include`/`exclude` pattern lists (`DB.SCHEMA.OBJECT`) must match excludes at the **same granularity** as includes — never collapse a fine-grained exclude to DB-level only. See `plugin-development` skill and `PLUGIN_DEVELOPMENT.md` for the canonical patterns.
- **DQL semantics — `dsoa.run.plugin` vs `dsoa.run.context`:** Use `dsoa.run.plugin` to filter all telemetry produced by a plugin (regardless of which context emitted it). Use `dsoa.run.context` only when targeting a specific named context within a plugin. Never use `dsoa.run.context == "<plugin_name>"` when the intent is to select by plugin — some plugins have multiple contexts and the filter will silently miss data. Example: shares events must use `dsoa.run.plugin == "shares"`, while logs from the inbound/outbound shares contexts use `in(dsoa.run.context, {"inbound_shares", "outbound_shares"})`.
- **Agent SQL call template sync:** Whenever a new plugin is added, its name must also be added to the `ARRAY_CONSTRUCT` block in the commented manual-call template in `src/dtagent.sql/agents/700_dtagent.sql`. The test `test/core/test_agent_registration.py` enforces this automatically.
- **Security:** Never commit credentials. Use `.gitignore` and `_snowflake.read_secret()`.
- **Backward compatibility:** Provide upgrade scripts for object changes. Document breaking changes.
- **Procedure signature changes:** Snowflake does not replace a procedure if the new signature would create an ambiguous overload with an existing one — it raises `"Cannot overload PROCEDURE ... as it would cause ambiguous PROCEDURE overloading"`. Whenever a stored procedure's parameter list changes (add, remove, or rename a parameter), **always** create an upgrade script in `src/dtagent.sql/upgrade/<new-version>/` that drops the old signature **before** the new one is deployed. Use `DROP PROCEDURE IF EXISTS DTAGENT_DB.APP.<NAME>(<old-types>)` wrapped in `--%PLUGIN:<name>:` / `--%:PLUGIN:<name>` guards. The upgrade script is assembled by `build.sh` into `build/09_upgrade/v<version>.sql` and applied via `./scripts/deploy/deploy.sh <env> --scope=upgrade --from-version=<prev-version> --options=skip_confirm` **before** the main `--scope=plugins,admin,config` deploy step.