GEMINI.md

# GEMINI.md

## Project Overview

This project, "Rust Expert | AI LLM," is dedicated to creating a highly specialized Large Language Model (LLM) with expertise in the Rust programming language. The primary goal is to develop a model that can answer questions, provide feedback, and generate high-quality, efficient, and idiomatic Rust code that is 100% executable.

The development strategy involves two main phases:
1.  **Supervised Fine-Tuning (SFT):** Fine-tuning a base model on a curated dataset of high-quality Rust code and text.
2.  **Direct Preference Optimization (DPO):** Further refining the SFT model using a preference dataset created by Rust experts.

The project is currently in the initial planning and environment setup phase (Phase 0). The detailed objectives and a comprehensive 8-week roadmap are documented in the `docs` directory.

## Retrieval-Augmented Generation (RAG)

To ensure the model provides the most current and factually accurate information, the project incorporates a Retrieval-Augmented Generation (RAG) architecture. Unlike training methods that internalize knowledge, RAG connects the LLM to an external, real-time knowledge source.

When a query is received, the RAG system first retrieves relevant information from a specialized database (e.g., the latest Rust documentation, crate specifications, or community tutorials). This retrieved context is then provided to the LLM along with the original prompt, enabling it to generate a response that is anchored in fresh, verifiable data. This approach drastically reduces "hallucinations" and allows the model to answer questions about very recent or specific topics without needing to be constantly retrained.

## Mixture of Experts (MoE)

To build a highly capable model while maintaining computational efficiency, this project will explore a Mixture of Experts (MoE) architecture. In an MoE model, multiple specialized "expert" sub-networks exist, and a gating network dynamically selects the most relevant experts to process each part of the input.

This approach allows the model to scale its parameter count significantly without proportionally increasing the computational cost for inference. For each token, only a fraction of the total parameters are activated, leading to faster and more efficient training and inference compared to dense models of equivalent size. This is particularly beneficial for creating a powerful and responsive Rust code generation assistant.

## Building and Running

The project is in its early stages, and the full build and run process is not yet established. However, the development environment will be based on Python.

### Environment Setup

1.  **Create a Python virtual environment:**
    ```bash
    python -m venv .venv
    source .venv/bin/activate
    ```

2.  **Install key dependencies:**
    ```bash
    pip install transformers datasets accelerate bitsandbytes peft trl jupyterlab
    ```

### Key Technologies

*   **Base Models:** Potential candidates include `codellama/CodeLlama-7b-hf`, `mistralai/Mistral-7B-v0.1`, and `bigcode/starcoder2-3b`.
*   **Training Libraries:** Hugging Face's `transformers`, `datasets`, `peft`, and `trl` libraries will be used for SFT and DPO.
*   **Hardware:** Prototyping will be done on `Google Colab Pro`, with final training on `Google Cloud AI Platform` using L4 or T4 GPUs.
*   **Monitoring:** `Weights & Biases` is recommended for monitoring training progress.

## Development Conventions

The project follows a structured and iterative development process outlined in the `docs/02-roadmap.md` file.

### Key Principles

*   **Executable Code:** A primary goal is to ensure that 99.99% of the code generated by the model is executable.
*   **Simplicity and Quality:** The model should favor practical, simple, and high-quality code, adhering to SOLID principles.
*   **Progressive Enhancement:** The model will be developed in stages, starting with basic capabilities and becoming more sophisticated with each iteration.

### Dataset Curation

*   **SFT Dataset:** High-quality `(instruction, response)` pairs will be collected from sources like GitHub, Stack Overflow, and "The Rust Programming Language" book.
*   **DPO Dataset:** A preference dataset will be created by generating multiple responses to challenging prompts and having experts choose the "winning" and "losing" responses.

### Experimentation

The project includes specific experiments to:
*   Validate the consistency of the S-FT process.
*   Compare the performance of different base models.