npx agentspec-cli install c5acfed3-5b79-4874-9cb5-47fdd19dd502 --file-type cursor-rules→ .cursor/rules/
---
description:
globs: *.yml,*.sh
alwaysApply: false
---
# Rules for .sh and ci.yml
This rule describes the Continuous Integration and Deployment (CI/CD) workflow for the Imageflow project, primarily defined in `.github/workflows/ci.yml` and utilizing helper scripts within the `ci/` directory.
## Key Aspects
- **Triggers:** Runs on pushes to any branch, pull requests, manual dispatch (`workflow_dispatch`), and automatically deploys on new GitHub Releases (`on.release.types: [published]`).
- **Matrix Builds:** Uses a comprehensive GitHub Actions matrix strategy (`jobs.build.strategy.matrix`) to build and test across various operating systems (Linux, macOS, Windows), architectures (x64, arm64, x86), and Rust targets (GNU, MSVC, MUSL), including cross-compilation via `houseabsolute/actions-rust-cross`.
- **Testing:**
- Executes `cargo test` for each matrix entry.
- Includes GLIBC version compatibility checks using `ci/check-glibc.sh` for specific Linux builds (`matrix.max-glibc`).
- Runs `imageflow_tool diagnose` for native (non-cross) builds.
- **Artifact Packaging:**
- Creates platform-specific archives (`.zip`/`.tar.gz`) containing binaries and licenses using `ci/pack_artifacts.sh`. Archives are stored in `./artifacts/github/` (for GitHub Releases) and `./artifacts/upload/` (for S3).
- Generates NuGet packages (`.nupkg`) using scripts in `ci/pack_nuget/`:
- `pack.sh`: Creates runtime-specific packages (`Imageflow.NativeRuntime.{rid}`, `Imageflow.NativeTool.{rid}`) for each build matrix entry (excluding static/MUSL builds). Output to `${REL_NUGET_OUTPUT_DIR}` (`artifacts/nuget/`). Uploaded as GitHub artifacts (`nuget-packages-{matrix.package-suffix}`).
- `pack_meta.sh`: Creates meta-packages (e.g., `Imageflow.NativeRuntime.All`, `Imageflow.Net.All`) in the separate `nuget-meta-packages` job.
- Builds Docker images (`imazen/imageflow_tool`, `imazen/imageflow_tool-arm64`) for MUSL targets using `docker/build-push-action`.
- **Deployment (Triggered by GitHub Release):**
- The `ALLOW_PUBLISH` environment variable controls deployment steps (`github.event_name == 'release'`).
- Uploads archives from `./artifacts/github/` to GitHub Releases.
- Uploads archives and potentially NuGet packages from `./artifacts/upload/` to AWS S3 (`$IMAGEFLOW_RELEASE_BUCKET`). Includes verification checks.
- Pushes Docker images to Docker Hub.
- The `nuget-meta-packages` job downloads all `.nupkg` artifacts from the `build` job runs, builds meta-packages, and then:
- Uploads all `.nupkg` files to S3 (`nuget/{version}/`).
- Publishes all `.nupkg` files to NuGet.org (controlled by `secrets.NUGET_UPLOAD_NUGET`).
- Publishes all `.nupkg` files to GitHub Packages (controlled by `secrets.NUGET_UPLOAD_GITHUB`).
- **Environment Variables:** Relies heavily on environment variables, many set dynamically based on the build matrix (e.g., `TARGET_DIR`, `TAG_SHA_SUFFIX`, `RUSTFLAGS`, `LIBIMAGEFLOW_DYNAMIC`, `ALLOW_PUBLISH`). Secrets are used for deployment credentials.
- **Conditional Logic:** Uses `if:` conditions extensively to control steps based on the matrix configuration (e.g., `matrix.cross`, `matrix.static`, `matrix.max-glibc`), event type (`github.event_name == 'release'`), and secrets.
## Key Scripts Invoked by `.github/workflows/ci.yml`
- `ci/check-glibc.sh`: Checks compiled library against maximum allowed GLIBC version.
- `ci/pack_artifacts.sh`: Packages binaries, licenses, etc., into `.zip`/`.tar.gz` archives for releases and S3.
- `ci/pack_nuget/pack.sh`: Creates runtime-specific NuGet packages.
- `ci/pack_nuget/pack_meta.sh`: Creates NuGet meta-packages.
- `ci/pack_nuget/upload_nuget.sh`: Uploads NuGet packages to nuget.org, handling potential failures and deletions.
- `ci/pack_nuget/tests/run_all_tests.sh`: Runs tests related to NuGet packaging early in the workflow.
*(Note: Other scripts exist within `ci/` subdirectories like `cloud/`, `nixtools/`, `wintools/`, `packaging_extras/` but are not directly called by the main steps in `.github/workflows/ci.yml`.)*
## Key Directories (Relative to Workspace Root)
| Directory | Purpose | Created By | Used By |
| :----------------------------- | :----------------------------------------------------------------------------------------------------- | :------------------------------------------ | :---------------------------------------------------------------------------------------------- |
| `./target/{target_triple}/` | Standard Cargo build output directory for a specific target. | `cargo build/test`, `cross build/test` | `cargo`, `cross`, `check-glibc.sh`, packaging scripts (`REL_BINARIES_DIR` depends on this) |
| `./target/{target_triple}/{profile}/` | Contains the actual compiled binaries (e.g., `release/`). (`$REL_BINARIES_DIR`) | `cargo build/test`, `cross build/test` | `check-glibc.sh`, `pack_artifacts.sh`, `pack_nuget/pack.sh`, `imageflow_tool diagnose`, Docker copy |
| `./artifacts/` | Root directory for storing intermediate and final packaged artifacts. | CI Steps (`mkdir`), Packaging scripts | Artifact upload/download steps |
| `./artifacts/github/` | Contains `.zip`/`.tar.gz` archives intended for GitHub Releases. | `pack_artifacts.sh` | `upload-to-github-release` action |
| `./artifacts/upload/` | Contains artifacts intended for S3 upload (archives, potentially NuGet). | `pack_artifacts.sh`, (`REL_NUGET_ARCHIVE_DIR`) | S3 upload action, Verification step |
| `./artifacts/nuget/` | Contains generated `.nupkg` files. (`$REL_NUGET_OUTPUT_DIR`) | CI Step (`mkdir`), `pack_nuget/*.sh` | Artifact upload/download, `upload_nuget.sh`, GitHub Packages upload |
| `./ci/` | Contains helper scripts for packaging, testing, and checks. | Repo | Various CI steps in `.github/workflows/ci.yml` |
| `./docker/imageflow_tool/` | Contains Dockerfile and context (copied binary) for building the tool image. | Repo, CI Step (copy binary) | Docker build steps |
## Core Assumptions
| Assumption Type | Details |
| :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Environment State** | Environment variables set using `echo "VAR=value" >> $GITHUB_ENV` in one step are available to all subsequent steps *within the same job*. Key variables like paths (`REL_BINARIES_DIR`), names (`IMAGEFLOW_TOOL`), and flags (`ALLOW_PUBLISH`) are established early and relied upon later. |
| **Filesystem Layout** | Code is checked out at the root. `cargo`/`cross` commands populate the `./target/` directory predictably. Scripts in `ci/` expect to find binaries in `$REL_BINARIES_DIR` and write outputs to specific `./artifacts/` subdirectories. Upload steps expect artifacts to exist in those locations. |
| **Script Execution** | Scripts in `ci/` are assumed to be executable (often preceded by `chmod +x` or assumed by `shell: bash`). They rely on specific environment variables being set by the workflow and may require specific command-line arguments. They assume standard tools (`bash`, `curl`, `sed`, `cargo`, `docker`, etc.) are available on the runner OS. |
| **Matrix Dependency** | The behavior of many steps (compilation flags, packaging, deployment eligibility, GLIBC checks) is heavily dependent on the variables defined in the `strategy.matrix` for the current job (`matrix.target`, `matrix.os`, `matrix.static`, `matrix.cross`, `matrix.max-glibc`, etc.). |
| **Build Output** | `cargo build` or `cross build` successfully produces the expected executables and libraries (`imageflow_tool[.exe]`, `libimageflow.so`/`.dll`/`.dylib`, `libimageflow.a`/`.lib`) in the location defined by `$REL_BINARIES_DIR`. Static MUSL builds produce `libimageflow.a` instead of a dynamic library. |
| **Release Trigger** | Deployment actions (uploading to releases, S3, Docker Hub, NuGet feeds) are primarily intended to run only when the workflow is triggered by publishing a GitHub Release (`github.event_name == 'release'`), controlled via the `ALLOW_PUBLISH` variable. |
## Key Environment Variables
| Variable | Purpose | Example Value (Conceptual) | Set By | Used By |
| :------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------- | :----------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PROFILE` | Cargo build profile. | `release` | Global `env` | Cargo commands (`build`, `test`, `doc`), `REL_BINARIES_DIR` definition |
| `REL_NUGET_OUTPUT_DIR` | Relative path to the directory where final `.nupkg` files are placed. | `artifacts/nuget/` | Global `env` | CI steps (`mkdir`), `pack_nuget/*.sh`, Artifact upload/download, `upload_nuget.sh`, GitHub Packages upload |
| `IMAGEFLOW_RELEASE_BUCKET` | AWS S3 bucket name for storing releases. | `imageflow-nightlies` | `vars` context or Global `env` | S3 upload action, Constructed URLs (`ESTIMATED_*_URL`) |
| `IMAGEFLOW_RELEASE_BUCKET_REGION` | AWS region for the release bucket. | `us-west-1` | `vars` context or Global `env` | Constructed URLs (`HTTPS_IMAGEFLOW_BUCKET`) |
| `matrix.target` | Rust target triple for compilation. | `x86_64-unknown-linux-gnu` | `strategy.matrix` | Cargo/Cross commands, `TARGET_DIR`, `REL_BINARIES_DIR`, Script logic (implicitly), Conditional steps (`contains(matrix.target, 'musl')`) |
| `matrix.os` | GitHub Actions runner OS identifier. | `ubuntu-22.04` | `strategy.matrix` | Job runner selection, `EXTENSION` logic, `IMAGEFLOW_TOOL` definition |
| `matrix.package-suffix` | Suffix used for naming artifacts and identifying build variants. | `linux-x64` | `strategy.matrix` | `TAG_SHA_SUFFIX`, `pack_artifacts.sh`, Artifact upload/verification steps, `pack_nuget/pack.sh`, GitHub Artifact Naming |
| `matrix.nuget-rid` | .NET Runtime Identifier corresponding to the build. | `linux-x64` | `strategy.matrix` | `pack_nuget/pack.sh` (for naming/metadata) |
| `matrix.cross` | Flag ('true'/'false') indicating if `cross` should be used for building/testing. | `true` | `strategy.matrix` | Conditional execution of `cargo` vs `cross` steps, `imageflow_tool diagnose` skip |
| `matrix.static` | Flag ('true'/'false') indicating if a static build (MUSL) is being performed. | `true` | `strategy.matrix` | NuGet packaging logic (skip runtime package artifact upload), Cargo.toml editing step (implicitly) |
| `matrix.max-glibc` | Maximum allowed GLIBC version for the build (if applicable). | `2.17` | `strategy.matrix` | Conditional execution of `check-glibc.sh` step |
| `matrix.docker-image-name` | Name of the Docker image to build/push (if applicable). | `imazen/imageflow_tool` | `strategy.matrix` | Conditional execution of Docker steps, `docker/metadata-action`, `docker/build-push-action` |
| `matrix.docker-platform` | Target platform for the Docker build. | `linux/amd64` | `strategy.matrix` | `docker/build-push-action` (`platforms` input) |
| `GITHUB_SHA_SHORT` | Short Git commit SHA. | `a1b2c3d` | `FranzDiebold/github-env-vars-action` | `TAG_SHA_SUFFIX` definition |
| `EXTENSION` | Archive file extension based on OS. | `.tar.gz` or `.zip` | `Set EXTENSION` step | `pack_artifacts.sh`, Artifact upload/verification steps (`ESTIMATED_*_URL`) |
| `TARGET_DIR` | Path to the target-specific build root directory. | `target/x86_64-unknown-linux-gnu/` | `Set env vars` step | `pack_artifacts.sh` (env input), `REL_BINARIES_DIR` definition |
| `REL_BINARIES_DIR` | Relative path to the directory containing compiled binaries for the current profile/target. | `target/x86_64-unknown-linux-gnu/release/` | `Set env vars` step | `check-glibc.sh`, `pack_artifacts.sh` (env input), `pack_nuget/pack.sh` (env input), `IMAGEFLOW_TOOL_PATH` definition, Docker copy step |
| `TAG_SHA_SUFFIX` | Combined identifier including ref, SHA, and package suffix. | `main-a1b2c3d-linux-x64` | `Set env vars` step | `IMAGEFLOW_TAG_SHA_SUFFIX` definition, `pack_artifacts.sh` (env input) |
| `CARGO_PROFILE_TARGET_FEATURES_ARGS` | Common arguments for Cargo commands (profile, target, features). | `--release --target=... --features=...` | `Set env vars` step | `cargo build`, `cargo test`, `cargo doc` |
| `CROSS_ARGS` | Common arguments for `cross` commands (profile, features). | `--locked --release --features=...` | `Set env vars` step | `cross build`, `cross test` (`args` input) |
| `ALLOW_PUBLISH` | Flag ('true'/'false') controlling whether deployment steps run (based on event type and matrix). | `true` or `false` | `Set env vars` step | Conditional execution of upload steps (GitHub Release, S3, Docker, NuGet via secrets) |
| `LIBIMAGEFLOW_DYNAMIC` | Platform-specific name of the dynamic library. | `libimageflow.so` | `Set env vars` step | `check-glibc.sh`, `pack_artifacts.sh` (env input) |
| `LIBIMAGEFLOW_STATIC` | Platform-specific name of the static library. | `libimageflow.a` | `Set env vars` step | `pack_artifacts.sh` (env input) |
| `IMAGEFLOW_TOOL` | Platform-specific name of the command-line tool. | `imageflow_tool` or `imageflow_tool.exe` | `Set env vars` step | `pack_artifacts.sh` (env input), `pack_nuget/pack.sh` (env input), `IMAGEFLOW_TOOL_PATH` definition |
| `IMAGEFLOW_NUGET_VERSION` | Version string for NuGet packages (tag name or 'vTEST'). | `v0.9.0` or `vTEST` | `Set more env vars` step | `pack_nuget/pack.sh` (env input), `pack_nuget/pack_meta.sh` (env input), `REL_NUGET_ARCHIVE_DIR` definition, S3 NuGet upload path |
| `IMAGEFLOW_TAG_SHA_SUFFIX` | Base name used for packaged artifacts. | `imageflow-main-a1b2c3d-linux-x64` | `Set more env vars` step | `pack_artifacts.sh` (env input), Artifact URL definitions (`ESTIMATED_*_URL`) |
| `IMAGEFLOW_TOOL_PATH` | Full path to the compiled `imageflow_tool` executable. | `target/.../release/imageflow_tool` | `Set more env vars` step | `imageflow_tool diagnose` step, Docker copy step |
| `ESTIMATED_ARTIFACT_URL*`, etc. | Constructed URLs used for verifying S3 uploads. | `https://s3.../imageflow-...tar.gz` | `Set more env vars` step | `Verify artifacts` step |
| `CI_TAG`, `CI_REPO`, etc. | Variables embedded into the binaries via `build.rs` for diagnostics. | `v0.9.0`, `imazen/imageflow` | `Set more env vars` step | `imageflow_core/build.rs` |
| `NUGET_UPLOAD_NUGET` | Flag ('true'/'false') controlling upload to nuget.org. Read from secrets. | `true` or `false` | `env` (from `secrets.NUGET_UPLOAD_NUGET`) | Conditional execution of `upload_nuget.sh` in `nuget-meta-packages` job, S3 NuGet meta upload |
| `NUGET_UPLOAD_GITHUB` | Flag ('true'/'false') controlling upload to GitHub Packages. Read from secrets. | `true` or `false` | `env` (from `secrets.NUGET_UPLOAD_GITHUB`) | Conditional execution of `curl` upload step in `nuget-meta-packages` job |
| `AWS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | Credentials for uploading to S3. | (Secret) | Secrets (`secrets.AWS_...`) | `s3-upload-action` |
| `NUGET_API_KEY` | API key for publishing to nuget.org. | (Secret) | Secrets (`secrets.NUGET_API_KEY`) | `upload_nuget.sh` (passed as argument) |
| `GITHUB_TOKEN` | Token for uploading to GitHub Releases and Packages. | (Secret) | `secrets.GITHUB_TOKEN` | `upload-to-github-release`, GitHub Packages `curl` command |
| `DOCKER_USERNAME`, `DOCKER_PASSWORD` | Credentials for logging into Docker Hub. | (Secret) | Secrets (`secrets.DOCKER_...`) | `docker/login-action` |
Here's a new idea. Let's completely migrate away from use of bash for nuget. We'll use dotnet tooling for everything, using properties for version, filename, properties, etc. We'll put everything that is the same between nuget packages in a shared .targets file in /dotnet/nuget/shared, and have a separate .csproj for every RID. native runtime and nativetool csproj can go in dotnet/nuget/native/ and meta csproj in dotnet/nuget/meta/. The meta csproj will have project references to the native projects, so they automatically resolve into nuget refs correctly when we build, pack, and push the solution (put it in /dotnet/nuget/Imageflow.sln) all together. We'll make sure that the artifacts from each runner are collected, keeping their rid subfolder, and then merged safely together into the final directory structure shared by all the NativeRuntime packages.
devops-automation
CI/CD pipeline design with GitHub Actions, Docker, Kubernetes, Helm, and GitOps patterns
by AgentHub Crawler
devops
CI/CD and DevOps workflow. Use when modifying GitHub Actions, Docker configurations, or compose files.
by AgentHub Crawler
ci-cd-pipelines
CI/CD pipeline patterns for GitHub Actions, GitLab CI, testing strategies, and deployment automation
by AgentHub Crawler
Sign in to leave a comment.
Git Workflow
Cursor IDE rules and conventions for AI-assisted development.
by AgentHub Crawler