MIP-0075 tasks¶
Ordered delivery of MIP-0075: the beach and water-quality
tables in the R2 DuckLake, beaches first (/marola-devkit:mip-tasks, stack). Branch
mip-0075/<k>-<slug> in the repo the row lands in; rows of one repo stack on that repo's previous
row and merge bottom-up. Each row's delivers cell starts with the repo its PR lands in (read by
tasks-to-issues). §n are the MIP's. Every marola-app row ends with
just build && just test && just quality green, every marola-oods row with just quality green,
and a row that changes behaviour updates that repo's README and docs/ in the same PR. No row
writes to the bucket from CI before the people's gates below are done.
| # | slug | delivers | tests (must exist before the PR) | depends on |
|---|---|---|---|---|
| 1 | oods-module | marola-app — the oods sbt module (dependsOn(local), in root.aggregate), cli dependsOn(oods) so the assembly carries it; org.duckdb:duckdb_jdbc 1.5.6.0 in oods only; marola.oods.Main printing §5.4's usage and exiting 2 on no or an unknown command |
MainSpec: no arguments and oods frobnicate exit 2 with the usage on stderr; sbt cli/assembly and jar tf lists marola/oods/Main.class (output in the PR) |
– |
| 2 | duckdb-extensions | marola-app — ducklake and httpfs for the pinned engine: an sbt task (duckdbExtensions, beside corpusFetch) fetching both files at a pinned version and sha256 into .tmp/duckdb/, which the tests load; the Dockerfile copies the same files into the image; the store loads them from files with autoinstall_known_extensions and autoload_known_extensions off. docker-smoke.yml gains a step that loads both offline (--network none) and attaches a local lake |
DuckDbExtensionsSpec: both load from .tmp/duckdb/ with autoinstall off and a ducklake: attach on a temp directory succeeds; a wrong sha256 fails the task. The smoke step green on the PR (run link) |
1 |
| 3 | oods-model | marola-app — marola.oods.model: BeachRow, FacilityRow, TrailRow, PointRow, SampleRow (MIP-0056 §5.2's plus agencyLabel, unit), each derives Schema (kyo-schema 1.0.0-RC7 in oods), opaque Uf, IbgeCode, AreaId, LatLon with smart constructors, and the enums of §5.2 (Indicator, Qualifier, CountUnit, Channel, GeoSource, SourceLevel, Mode, RunOutcome; Facility gains label/fromLabel) |
OodsModelSpec: for every enum, fromLabel(label(x)) == Some(x) over values and fromLabel("x") == None; Uf("sc"), IbgeCode("420540"), AreaId("Floripa") and a LatLon outside Brazil's box are rejected |
1 |
| 4 | lake-sql | marola-app — OodsS3Config from the OODS_* settings of §5.4 (redacted toString, a missing one is a typed error); lake-contract.version pinning a marola-oods tag, and scripts/lake-contract-fetch.sh (just lake-contract-fetch, sbt's lakeContractFetch beside corpusFetch) unpacking its marola-oods-lake-<tag>.tar.gz (migrations/, views.sql, checks.sql) into .tmp/lake-contract, which the tests read and the Dockerfile copies into the image; never a copy of marola-oods's tree |
OodsS3ConfigSpec: toString never contains the secret; each missing required setting is named in the error. LakeSqlSpec: the fetched migrations, views.sql and checks.sql run in a local DuckLake; with point_fitness broken to count unknowns, checks.sql fails with US3.1; for each row class of row 3, Structure.of[Row] (snake_case names; Option ↔ nullable; DuckDB type by Tag, since kyo-schema reports LocalDate/LocalTime as String) equals that table's information_schema.columns, and a field renamed in a scratch copy fails it |
1, 2, 3, 22 |
| 5 | lake-store | marola-app — OodsStore (§5.4) and DuckLakeStore: session TYPE s3 secret from OodsS3Config (never PERSISTENT), ATTACH 'ducklake:$OODS_CATALOG' with DATA_PATH and DATA_INLINING_ROW_LIMIT 0; on attach, the contract's pending migrations applied by row 22's rules (each in one transaction with its schema_migration row and md5; a duplicate version or an applied file whose md5 changed refused before anything runs; views.sql re-applied only when its md5 differs from the version-0 row), so an empty catalog becomes the full schema and a current one gets no snapshot; upsertArea and upsertPartition as §5.4's three statements in one transaction after oods check; openRun/closeRun, knownPartitions, maintain; one connection in a Kyo Scope, every call in Sync.defer |
DuckLakeStoreSpec on a local lake, §7's cases: an identical load makes no snapshot; time travel returns the old rows; a batch failing oods check rolls back and writes nothing; a gone key is deleted; samples are never deleted; maintain expires snapshots older than its window; a READ_ONLY attach refuses a write; a catalog built by marola-oods's lake-migrate.sh from the pinned tag attaches and loads with no DDL run; an empty catalog gets every table; an edited migration and a duplicate version are refused; a changed views.sql is applied once |
3, 4 |
| 6 | lake-store-it | marola-app — DuckLakeStoreIT: §7's integration suite, MinIO in Testcontainers, tag Integration, excluded from just test; a CI job on ubuntu-latest running it |
The IT itself, green in that job: row 5's cases with DATA_PATH on S3, plus the catalog closed and re-attached between loads (run link in the PR). actionlint |
5 |
| 7 | contract-ci | marola-oods — oods-check.yml runs lake/checks.sql on the DuckDB CLI pinned in flake.nix plain (ci.yml's lake-migrate.sh --self-test already runs it inside a local DuckLake); just checks-sql runs the same |
Green on the PR; a commit on a scratch branch breaking point_fitness makes the job red with US3.1 (run link) |
22 |
| 8 | areas-file | marola-oods — etl/areas.json (id, lat, lon, radius_km, beach_limit for floripa, rio, salvador, copied from marola-site's site/areas.json) and scripts/etl-inputs-check.sh checking its shape, run by just quality |
etl-inputs-check.sh --self-test: a missing field, a duplicate id and a latitude outside Brazil each fail; the real file passes |
– |
| 9 | beach-load | marola-app — BeachLoad and oods beaches: per area, BeachFinder.nearby(snapshots = None), OverpassAccessibilityClient.near, TrailFinder.nearby → rows → upsertArea → fetch_run; partial when only facilities or trails fail; SuspiciousShrink under half the stored beaches; --dry-run attaches no lake; the stderr status line; exit codes 0–3 |
BeachLoadSpec from captured Overpass answers through Http.withTransport: exact rows; run twice → unchanged, no new snapshot; every mirror 504 → failed, rolled back, fetch_run says so; trails 429 → partial, beaches written; a half-size answer refused |
5 |
| 10 | lake-export | marola-app — oods maintain and oods export: exports/beaches/<BeachSnapshot.key>.json per area via BeachSnapshot.encode, exports/water-quality/<source>.json in CachedWaterQualityClient's v1 format from latest_per_point (empty until row 16 loads samples), --water-positions mirrored into water_position first; oods status |
LakeExportSpec: a loaded area's export decodes with BeachSnapshot.decode and BeachFinder reads it from MAROLA_BEACHES_DIR without a request; a fixture sample's export round-trips through CachedWaterQualityClient and carries its water position; a second export is byte-identical |
9 |
| 11 | beach-workflow | marola-oods — beach-etl.yml per §5.4: Monday cron, dispatch with area and dry_run, concurrency: oods-lake, the catalog download (an empty listing starts a new catalog, which DuckLakeStore migrates on attach from the pinned lake contract; a refused one stops) / oods beaches / maintain / upload (always) / export steps, the catalog steps as a composite action in .github/actions/oods-lake/, CLOUDFLARE_R2_* mapped to OODS_*; marola-image bumped to the first image with marola.oods.Main; README, AGENTS.md and docs/3-development.md say the workflows write the bucket, never this repo, and name their secrets and variables |
actionlint. A dry_run dispatch on the branch, run by a person, is green and writes nothing (run link). After merge, a person dispatches area=all twice: the first creates the catalog at the contract's version, the second prints changed=0 snapshot=- (run links, rows, catalog size and seconds in the PR) |
8, 10, 22 |
| 12 | planner | marola-app — Planner (pure): partitions per source, state and year; the mutability rule of §5.4; immutable partitions with a matching hash in knownPartitions skipped; an unknown UF is an error |
PlannerSpec: mode × date × known-partition tables, including the January rule at 2027-01-10 and its absence at 2027-02-20; an unknown UF fails |
3 |
| 13 | throttle | marola-app — Throttle: ≥ 250 ms per host, ≤ 4 in flight, 3 attempts with backoff on 5xx and time-outs, stop on 429/403 |
ThrottleSpec with a fake clock and a recording Http.Transport: spacing per host; a 500 then 200 succeeds in two attempts; three 500s fail; a 429 stops before the next request |
1 |
| 14 | ima-sc-adapter | marola-app — ImaScAdapter: incremental from /relatorio/mapa (reusing ImaScWaterQualityClient.parse's field handling), backfill from the CSV per beach-year, points from the feed; e_coli, NMP/100mL, <20 → (20, below); fixtures already in oods/src/test/resources/ima-sc/ |
ImaScAdapterSpec: MIP-0056.tasks row 2's cases (2003: 31 rows, 2010: 29, 2025: 145, 2026: 90; the censored value; both empty shapes; every Campeche row resolves to one of five UUIDs; an invented point name keeps a slug key with geo_source = none) |
12, 13 |
| 15 | sources-positions | marola-oods — etl/sources.json (ima-sc only: institute, state, level, channel, cron, brazil_only) and etl/water-positions.csv (header only), both checked by etl-inputs-check.sh |
etl-inputs-check.sh --self-test adds: a source with an unknown channel fails; a position row with one of three fields set fails; one outside Brazil's box fails |
8 |
| 16 | water-load | marola-app — Load and oods load: plan → fetch through Throttle → adapter rows → upsertPartition with its fetch_partition row → fetch_run; --dry-run; --max-minutes → partial; the source table mirrored from --sources; per-source status line; exit codes |
LoadSpec on a local lake: a second run changes no row and makes no snapshot; a 5xx → failed, rolled back; a 429 stops without a retry; a backfill killed after two partitions resumes with the third |
5, 14 |
| 17 | water-workflow | marola-oods — water-quality-etl.yml per §5.4, SC only: the plan job, the load matrix with max-parallel: 1, concurrency: oods-lake, row 11's oods-lake composite action, timeout-minutes: 330; marola-image bumped to an image with row 16 |
actionlint. A dry_run dispatch for SC, run by a person, is green (run link). After merge, a person dispatches the SC backfill, then the weekly run twice: requests, partitions, MB and minutes in the PR, the second run no_new_bulletin |
11, 15, 16 |
| 18 | inea-rj-adapter | marola-app — IneaRjAdapter: city pages → zones from the last 45 days → PDFs → IneaPdfParser; curated coordinates; uncurated codes kept with geo_source = 'none'; enterococci; the filename date |
IneaRjAdapterSpec from captured pages and a zone PDF to exact rows, an uncurated code kept, not dropped |
16 |
| 19 | inema-ba-adapter | marola-app — InemaBaAdapter: the newest campaign (not the pinned 83453), InemaPdfParser, curated coordinates; no sample date presented as a lab date |
InemaBaAdapterSpec from a captured campaign list and PDF to exact rows; the campaign picked is the newest in the fixture |
16 |
| 20 | rj-ba-workflow | marola-oods — inea-rj and inema-ba in etl/sources.json (brazil_only: true), the Friday and Saturday crons, scripts/br-proxy.sh from marola-dev/marola-site#20 started before a brazil_only state, MAROLA_BR_PROXY passed only then; marola-image bumped |
actionlint; etl-inputs-check.sh green. After merge and once the proxy exists, a person dispatches RJ and BA (run links) |
17, 18, 19 |
| 21 | mip-implemented | marola — MIP-0075 flipped to Implemented with every PR number | mip_graph.py --check; docs-lint green |
6, 7, 11, 17, 20, 22 |
| 22 | lake-schema | marola-oods — schema first: lake/migrations/0001_init.sql (§5.2's nine tables, sample partitioned by source_id, and schema_migration(version, name, checksum, applied_at)), lake/views.sql and lake/checks.sql (moved from specs/001-beach-persistence/contracts/); scripts/lake-migrate.sh applying them to a local lake only (just lake-migrate, run by just quality): each migration one transaction with its row and md5, a duplicate version or an edited applied migration refused, views.sql's md5 as the version-0 row; scripts/lake-contract.sh and a v*-tag release.yml attaching marola-oods-lake-<tag>.tar.gz; the DuckDB CLI pinned in flake.nix. Nothing here writes the bucket. On top of marola-dev/marola-oods#3 |
lake-migrate.sh --self-test: an empty local lake gets every table, the partitioning and schema_migration = 1 with both md5s; views.sql and checks.sql run on it; a second run applies nothing and makes no snapshot; a broken migration rolls back and records nothing; a duplicate version and an edited migration are refused; a changed views.sql is applied once. lake-contract.sh --self-test: the tarball is byte-identical after a touch and a chmod |
– |
The people's gates¶
Not rows: no agent can do them, and none blocks writing or reviewing a row's code.
- Before row 4: marola-dev/marola-oods#22 merged, and a person tags marola-oods
v0.1.0sorelease.ymlattaches the firstmarola-oods-lake-v0.1.0.tar.gz. - Before row 11's first real dispatch: §11's Phase 2 exception; §5.6's bucket, ETL token, marola-oods secrets and smoke test.
- Before row 20's dispatch: the Brazil proxy VM and
MAROLA_BR_PROXY, with its cost confirmed (marola-dev/marola-site#20). - Every dispatch of a workflow is a person's (marola-oods
AGENTS.md).
Decisions¶
- This list replaces MIP-0056.tasks rows 1–8 for the store. MIP-0056 wrote raw files and
Parquet into git on marola-oods's
main; this MIP keeps its adapters, planner, columns and fixtures and writes the lake instead (§5.1). Row 14's tests are MIP-0056.tasks row 2's. MIP-0056's PDF channel (its row 7) stays a later row of its own. - The site's read path is not here.
site.yml's download ofexports/with the read-only token (§5.5) is marola-site's own task, filed there once row 11's exports exist. - Facility and trail snapshot readers (so a build makes no Overpass call at all) are a follow-up in marola-app after row 10, not part of creating the tables.
- Rows 1–11 and 22 are tracked in marola-oods (the maintainer, 2026-10-06), whichever repo the PR
lands in, under the parent marola-dev/marola#672; a marola-app PR closes its row with
Closes marola-dev/marola-oods#N. - Schema first (the maintainer, 2026-10-05; revised 2026-10-06 on marola-dev/marola-oods#22): the tables are numbered migrations in marola-oods's
lake/, next to the views and checks, released by tag asmarola-oods-lake-<tag>.tar.gz. marola-app pins the tag (row 4) andDuckLakeStoreapplies only what is pending, on attach inside anoods-lakejob (row 5), so the bucket's catalog is created by the firstbeach-etl.ymlrun and never written from a laptop. The schema has one home, the SQL; a later change is a newNNNN_*.sql, never a rebuilt lake. The Scala row types are checked against it (row 4), not generated from it:kyo-schema1.0.0-RC7 has no DDL generator andkyo-sqlno DuckDB backend. Row 22 is listed last to keep the filed issue numbers; it runs first. - Optional later: a backfill that re-dispatches itself with
GITHUB_TOKEN, capped in hops.