Skip to content

MIP-0010 tasks

Ordered delivery of MIP-0010 as stacked PRs (.claude/skills/mip-tasks/SKILL.md, scripts/stack.sh). Branch mip-0010/<k>-<slug>; merge bottom-up within a lane, scripts/stack.sh restack after each squash-merge.

Scope is v1 only: the local mlflow server path (MIP Phase 0). The hosted tracking-server path (MIP §4.5, Phase 2) is out of scope for every task below: it needs Phase 1 (the Telegram bot, MIP-0002) done first per AGENTS.md's phase-discipline rule, and the human go-ahead the cost rule requires before any paid cloud resource, so there is nothing to build here yet regardless.

Revised while building task 6 (2026-09-05): the tracing lane is now stacked on the ledger lane: task 5 branches from task 4, task 6 from task 5, because MlflowTracing needs two things the ledger lane owns: MAROLA_MLFLOW_TRACKING_URI/MAROLA_MLFLOW_EXPERIMENT in AppConfig (task 3) and the experiment name → id lookup (MlflowApi, extracted from task 2's ledger), since MLflow's OTLP endpoint takes an experiment id header. Task 7 (pure Python, no code dependency) is stacked on task 6 as well, purely so the stack tooling's numbered-chain convention holds (scripts/stack.sh/uprds.sh derive task k's base as task k-1); merge order is 1 → 2 → 3 → 4 → 5 → 6 → 7. The paragraph below is kept as the original plan.

Three independent lanes, not one chain: the table's numbering is delivery order within a lane, not a single dependency chain across all seven. Real code dependency:

  • Ledger lane (1 → 2 → 3 → 4): each based on the previous, per scripts/stack.sh's normal numbered-branch convention.
  • Tracing lane (5 → 6): touches Telemetry/AppConfig/cli/Main, never RunLedger; task 5 branches from main directly (not from task 4), task 6 from task 5.
  • DSPy lane (7): pure Python, zero Scala dependency; branches from main directly.

The three lanes can be built in parallel (three fresh sessions/agents at once, per the skill's "one task, one session" step). Expect one small, mechanical conflict where lanes overlap in AppConfig.scala (ledger adds MAROLA_MLFLOW_TRACKING_URI/MAROLA_MLFLOW_EXPERIMENT, tracing adds MAROLA_TRACES); whichever lane merges second resolves it on restack, not a redesign.

# slug delivers tests (must exist before the PR) depends on
1 run-ledger-core core/ledger/RunLedger.scala: the trait (start/metrics/artifact/end), RunHandle, RunLedger.Noop; MIP → Accepted, this task list NoopRunLedgerSpec: every method returns unit, no transport call is possible (no transport exists yet) –
2 mlflow-run-ledger local/ledger/MlflowRunLedger.scala: REST over Http/Json, experiments/get-by-name→create, runs/create, runs/log-batch (chunked ≤1000 metrics/≤100 params per call, keys truncated to 250 chars with a warning), runs/update (RUNNING→FINISHED/FAILED), artifact upload via the server's artifact proxy MlflowRunLedgerSpec: exact JSON of runs/create and runs/log-batch via a scripted Http.Transport; chunking at the caps; 250-char truncation; FAILED status on end(ok = false) 1
3 mlflow-compose docker-compose.yml profile mlflow (ghcr.io/mlflow/mlflow, pinned tag; SQLite backend-store + .tmp/mlflow/ artifacts, --serve-artifacts, bound to 127.0.0.1); just mlflow-up/just mlflow-ui; AppConfig: MAROLA_MLFLOW_TRACKING_URI (unset ⇒ RunLedger.Noop), MAROLA_MLFLOW_EXPERIMENT (default marola) selecting MlflowRunLedger; RUN-LOCALLY.md "the MLflow ledger" walkthrough (MIP §3's transcript) docker compose --profile mlflow config --quiet added to just quality's existing compose-config check; live: just mlflow-up reaches 127.0.0.1:5000 2
4 benchmark-ledger OceanBenchmark.run logs to the configured RunLedger: experiment marola/benchmark; params model, embed_model, min_score, corpus_sha (hash of knowledge/*.md), git_sha, questions; metrics per arm (.coverage_in_corpus, .coverage_general, .coverage_all, .cited_pct, .abstained_pct, .mean_ms); artifact: the Markdown report. The Markdown report stays the canonical benchmark_gate.py input (MIP §5); this is additive, not a replacement BenchmarkLedgerSpec: the params/metrics map built from an OceanBenchmark.Report, asserted against a RunLedger test double; live: just mlflow-up && MAROLA_MLFLOW_TRACKING_URI=… just benchmark shows the run and its artifact in the UI 3
5 tracing-core-split Telemetry splits into core/Tracing.scala (withSpan, llmSpan trait) + core/Tracing.Noop; AppConfig: MAROLA_TRACES=off|mlflow TracingConfigSpec: every MAROLA_TRACES value (including unset) resolves to the right backend, pure function, no network – (branches from main, tracing lane)
6 mlflow-tracing local/MlflowTracing.scala (OTLP/HTTP exporter to <tracking uri>/v1/traces, header x-mlflow-experiment-id from MAROLA_MLFLOW_EXPERIMENT; pin the OpenTelemetry BOM to one version, MIP §4.3); TracedLlmClient(inner, tracing) decorator, one span per complete with gen_ai.request.model, latency, token counts (only when the response carries usage); prompt/completion text attached only when MAROLA_TRACE_CONTENT=1; wraps LocalLlmClient in cli/Main; RUN-LOCALLY.md traces walkthrough, ARCHITECTURE.md §5f updated TracedLlmClientSpec: span name, gen_ai.* attributes, content-off-by-default, using opentelemetry-sdk-testing's in-memory exporter; live: just mlflow-up && MAROLA_TRACES=mlflow just run -- --summarize shows one trace, two spans 5
7 dspy-mlflow-logging dspy/compile_recommendation_prompt.py logs to MLflow (Python client): experiment marola/prompt-compile, params (model, optimiser, trainset size), metrics (the metric's score per compiled program), artifacts (both prompt JSONs); resolves MIP §11 OQ3 (mlflow.dspy.autolog(): use it if it exists at the pinned versions, else log the two scores by hand) and OQ4 (keep the existing Langfuse hook alongside MLflow for v1, additive, remove only if it turns out redundant); FUTURE-WORK.md §10 marks the Langfuse-shaped JVM gap closed by tasks 5-6 ruff (just quality); a recorded-fixture self-test in the style of scripts/smoke_record.py --self-test, asserting the params/metrics dict built from a compile run; live: one real compile logs a run with two artifacts – (branches from main, dspy lane)

Decisions taken for v1 (MIP §11), so the tasks above are unambiguous:

  1. Local server only, via compose, not nixpkgs (OQ2): ghcr.io/mlflow/mlflow in a docker-compose.yml profile, never added to flake.nix's default shell or the runtime Dockerfile (MIP §8: "a heavy Python dependency"). Task 3 pins the exact tag it first uses.
  2. REST over Http/Json, not the Java client (MIP §4.2's own pick, restated): task 2's MlflowRunLedgerSpec is the fallback trigger: if artifact upload over REST proves awkward there, swap to org.mlflow:mlflow-client for that one call, not the whole ledger.
  3. Experiment naming (OQ6): marola/<kind> as already used throughout MIP §5: marola/benchmark, marola/prompt-compile, marola/finetune (later, unbuilt), marola/traces.
  4. Hosted tracking-server auth (OQ1) and self-hosting MLflow (MIP §4.5) are not tasks here — Phase 2, blocked on Phase 1 and the cost-confirmation gate regardless of MIP status.
  5. finetune/train_lora.py logging (MIP §5's fourth bullet) is not a task here: that script is "written, not run" today (MIP-0008); add its MLflow logging alongside whenever it is actually run for the first time, not speculatively.

Restack notes: task 3 is the first to touch docker-compose.yml and AppConfig (ledger lane); task 5 renames Telemetry call sites (cli/Main, wherever Http/pipeline code references it today): grep for every caller before moving code, not just the trait. Task 6 touches the same cli/Main wiring task 5 does (the TracedLlmClient wrap point); expect a small conflict there on restack, not a redesign. Whichever of task 3 or task 5 merges second conflicts trivially in AppConfig.scala (two independent env-var additions, not overlapping logic); resolve on restack. Nothing else rewrites a file an earlier task created.