Skip to content
← All projects
Project system

PromptPilot

Prompt versioning and A/B testing across AI providers.

View on GitHub →

What it does

PromptPilot creates prompt YAML files, saves version backups, diffs prompt revisions, and runs provider-backed A/B tests from a CLI.

System map

  1. 1

    Prompt YAML

    Template, variables, metadata

  2. 2

    Version Store

    Backups and history

  3. 3

    Diff

    Git or file history

  4. 4

    Providers

    OpenAI, Claude, local models

  5. 5

    A/B Result

    Cost, response, winner

Prompt YAML

Template, variables, metadata

Codebase read

  • Prompt files are stored as YAML with metadata, variables, token estimates, and version history.
  • Versioning supports manual backups and git diff fallback when a repository is present.
  • The CLI lazy-loads heavier dependencies so common commands start quickly.

Key snippet

promptpilot init my-summary \
  --description "Summarize text in 3 paragraphs"

promptpilot abtest my-summary \
  --input sample.txt \
  --provider openai

Tech used

PythonClickYAMLGitPythonOpenAIAnthropicTransformers

Source paths inspected

src/promptpilot/cli.pyversioning.pyutils.py

More project systems

GoVisualZero-config HTTP request visualizer for Go applications.→
FastAPI RadarReal-time request, SQL query, exception, and trace dashboard for FastAPI apps.→
LLMDogTerminal UI for selecting code context and turning it into LLM-ready Markdown.→
GIQAI-enhanced Git workflow assistant that keeps normal Git semantics intact.→