py-metrics

Architecture

PyMetrics is a thin FastAPI façade over a Celery-driven analysis pipeline. SQLite is authoritative for runs + metrics, Redis is the broker + result backend, and MinIO is the artifact store. The frontend is a static React build served by nginx.

Service graph

flowchart LR
  user([Browser]) -->|HTTP| fe[Frontend<br/>nginx + React]
  fe -->|REST + SSE| be[Backend<br/>FastAPI]
  be -->|enqueue| redis[(Redis)]
  redis -->|broker| worker[Celery worker]
  worker -->|metrics| db[(SQLite<br/>/data/db)]
  worker -->|ZIP/CSV| minio[(MinIO)]
  be -->|read| db
  be -->|presign| minio
  worker -. clone .-> github[(GitHub)]

Every arrow is synchronous HTTP except redis ↔ worker, which is the Celery broker protocol.

Analysis pipeline (one run)

sequenceDiagram
  participant API as FastAPI
  participant W as Celery worker
  participant FS as /data/repos
  participant DB as SQLite
  participant S3 as MinIO

  API->>DB: INSERT Analysis(status=pending)
  API->>W: enqueue run_analysis(id)
  W->>FS: git clone --depth 1 <url>
  W->>W: discover *.py files
  loop per file
    W->>W: radon + pylint + AST + deps
  end
  W->>W: pydriller (if shallow=false)
  W->>DB: bulk insert file/class/function metrics
  W->>W: generate CSVs + ZIP under descriptive output prefix
  W->>S3: upload ZIP + CSVs when MinIO/S3 is enabled
  W->>DB: UPDATE status=done
  W->>FS: rm -rf <clone>

Directory layout

py-metrics/
├── backend/                  # FastAPI + Celery + SQLAlchemy
│   ├── app/
│   │   ├── routers/         # HTTP routes (analysis, metrics, download, auth)
│   │   ├── services/        # radon, pylint, git, AST, CSV, S3, cache, auth
│   │   ├── tasks/           # celery_app + analysis_task + file_processor
│   │   ├── models/          # SQLAlchemy declarative models
│   │   ├── schemas/         # Pydantic v2 request/response models
│   │   ├── config.py        # Settings (pydantic-settings)
│   │   └── main.py          # FastAPI application wiring
│   ├── Dockerfile           # Multi-stage, non-root UID 1000
│   └── requirements.txt
├── frontend/                 # React 18 + Vite + Bulma
├── dataset/                  # Kaggle-ready dataset pipeline
├── docs/                     # MkDocs Material (this site)
├── data/                     # Bind-mount: db/, repos/, outputs/
└── docker-compose.yml

Data model

Four metric tables persist results, parented by analyses.id:

• analyses — one row per run, holds status, repo URL, requested ref, commit, and timings.
• file_metrics — one row per Python file with size, complexity, maintainability, Halstead, Git, Pylint, and dependency columns.
• class_metrics — one row per class (CK / OO metrics and class-level CC aggregates).
• function_metrics — one row per function (CC + Halstead + AST).
• Project summaries are computed on demand by the summary endpoint and exported into metrics_summary.csv.

The metrics.db SQLite file lives under /data/db/ in the container, mapped to ./data/db/ on the host. Generated exports live under descriptive folders in ./data/outputs/, such as flask__20260503-002114__ref-main__a3__sha-1a2b3c4d5e6f/. A separate migration workflow is not in place yet — schema evolves via SQLAlchemy create_all() on startup.

Why this layout

• FastAPI + Celery splits HTTP latency from CPU-bound radon/pylint/pydriller work. CELERY_WORKER_CONCURRENCY controls how many repositories run at once, while PROCESS_WORKERS controls per-repository file-processing threads.
• SQLite is sufficient at the expected volume (hundreds of runs) and trivializes persistence across dev machines. Swapping to Postgres requires only a DATABASE_URL change.
• MinIO decouples artifact storage from the backend filesystem — production deployments can point S3_ENDPOINT_URL at AWS/GCS/R2 without code changes.
• Shallow clone by default — git-history metrics are the slowest to compute; most dataset consumers care about structural metrics first. Set CLONE_SHALLOW=false to opt back in.

This site is open source. Improve this page.