Skip to content

MIP-0075 tasks

Ordered delivery of MIP-0075: the beach and water-quality tables in the R2 DuckLake, beaches first (/marola-devkit:mip-tasks, stack). Branch mip-0075/<k>-<slug> in the repo the row lands in; rows of one repo stack on that repo's previous row and merge bottom-up. Each row's delivers cell starts with the repo its PR lands in (read by tasks-to-issues). §n are the MIP's. Every marola-app row ends with just build && just test && just quality green, every marola-oods row with just quality green, and a row that changes behaviour updates that repo's README and docs/ in the same PR. No row writes to the bucket from CI before the people's gates below are done.

# slug delivers tests (must exist before the PR) depends on
1 oods-module marola-app — the oods sbt module (dependsOn(local), in root.aggregate), cli dependsOn(oods) so the assembly carries it; org.duckdb:duckdb_jdbc 1.5.6.0 in oods only; marola.oods.Main printing §5.4's usage and exiting 2 on no or an unknown command MainSpec: no arguments and oods frobnicate exit 2 with the usage on stderr; sbt cli/assembly and jar tf lists marola/oods/Main.class (output in the PR) –
2 duckdb-extensions marola-app — ducklake and httpfs for the pinned engine: an sbt task (duckdbExtensions, beside corpusFetch) fetching both files at a pinned version and sha256 into .tmp/duckdb/, which the tests load; the Dockerfile copies the same files into the image; the store loads them from files with autoinstall_known_extensions and autoload_known_extensions off. docker-smoke.yml gains a step that loads both offline (--network none) and attaches a local lake DuckDbExtensionsSpec: both load from .tmp/duckdb/ with autoinstall off and a ducklake: attach on a temp directory succeeds; a wrong sha256 fails the task. The smoke step green on the PR (run link) 1
3 oods-model marola-app — marola.oods.model: BeachRow, FacilityRow, TrailRow, PointRow, SampleRow (MIP-0056 §5.2's plus agencyLabel, unit), each derives Schema (kyo-schema 1.0.0-RC7 in oods), opaque Uf, IbgeCode, AreaId, LatLon with smart constructors, and the enums of §5.2 (Indicator, Qualifier, CountUnit, Channel, GeoSource, SourceLevel, Mode, RunOutcome; Facility gains label/fromLabel) OodsModelSpec: for every enum, fromLabel(label(x)) == Some(x) over values and fromLabel("x") == None; Uf("sc"), IbgeCode("420540"), AreaId("Floripa") and a LatLon outside Brazil's box are rejected 1
4 lake-sql marola-app — OodsS3Config from the OODS_* settings of §5.4 (redacted toString, a missing one is a typed error); lake-contract.version pinning a marola-oods tag, and scripts/lake-contract-fetch.sh (just lake-contract-fetch, sbt's lakeContractFetch beside corpusFetch) unpacking its marola-oods-lake-<tag>.tar.gz (migrations/, views.sql, checks.sql) into .tmp/lake-contract, which the tests read and the Dockerfile copies into the image; never a copy of marola-oods's tree OodsS3ConfigSpec: toString never contains the secret; each missing required setting is named in the error. LakeSqlSpec: the fetched migrations, views.sql and checks.sql run in a local DuckLake; with point_fitness broken to count unknowns, checks.sql fails with US3.1; for each row class of row 3, Structure.of[Row] (snake_case names; Option ↔ nullable; DuckDB type by Tag, since kyo-schema reports LocalDate/LocalTime as String) equals that table's information_schema.columns, and a field renamed in a scratch copy fails it 1, 2, 3, 22
5 lake-store marola-app — OodsStore (§5.4) and DuckLakeStore: session TYPE s3 secret from OodsS3Config (never PERSISTENT), ATTACH 'ducklake:$OODS_CATALOG' with DATA_PATH and DATA_INLINING_ROW_LIMIT 0; on attach, the contract's pending migrations applied by row 22's rules (each in one transaction with its schema_migration row and md5; a duplicate version or an applied file whose md5 changed refused before anything runs; views.sql re-applied only when its md5 differs from the version-0 row), so an empty catalog becomes the full schema and a current one gets no snapshot; upsertArea and upsertPartition as §5.4's three statements in one transaction after oods check; openRun/closeRun, knownPartitions, maintain; one connection in a Kyo Scope, every call in Sync.defer DuckLakeStoreSpec on a local lake, §7's cases: an identical load makes no snapshot; time travel returns the old rows; a batch failing oods check rolls back and writes nothing; a gone key is deleted; samples are never deleted; maintain expires snapshots older than its window; a READ_ONLY attach refuses a write; a catalog built by marola-oods's lake-migrate.sh from the pinned tag attaches and loads with no DDL run; an empty catalog gets every table; an edited migration and a duplicate version are refused; a changed views.sql is applied once 3, 4
6 lake-store-it marola-app — DuckLakeStoreIT: §7's integration suite, MinIO in Testcontainers, tag Integration, excluded from just test; a CI job on ubuntu-latest running it The IT itself, green in that job: row 5's cases with DATA_PATH on S3, plus the catalog closed and re-attached between loads (run link in the PR). actionlint 5
7 contract-ci marola-oods — oods-check.yml runs lake/checks.sql on the DuckDB CLI pinned in flake.nix plain (ci.yml's lake-migrate.sh --self-test already runs it inside a local DuckLake); just checks-sql runs the same Green on the PR; a commit on a scratch branch breaking point_fitness makes the job red with US3.1 (run link) 22
8 areas-file marola-oods — etl/areas.json (id, lat, lon, radius_km, beach_limit for floripa, rio, salvador, copied from marola-site's site/areas.json) and scripts/etl-inputs-check.sh checking its shape, run by just quality etl-inputs-check.sh --self-test: a missing field, a duplicate id and a latitude outside Brazil each fail; the real file passes –
9 beach-load marola-app — BeachLoad and oods beaches: per area, BeachFinder.nearby(snapshots = None), OverpassAccessibilityClient.near, TrailFinder.nearby → rows → upsertArea → fetch_run; partial when only facilities or trails fail; SuspiciousShrink under half the stored beaches; --dry-run attaches no lake; the stderr status line; exit codes 0–3 BeachLoadSpec from captured Overpass answers through Http.withTransport: exact rows; run twice → unchanged, no new snapshot; every mirror 504 → failed, rolled back, fetch_run says so; trails 429 → partial, beaches written; a half-size answer refused 5
10 lake-export marola-app — oods maintain and oods export: exports/beaches/<BeachSnapshot.key>.json per area via BeachSnapshot.encode, exports/water-quality/<source>.json in CachedWaterQualityClient's v1 format from latest_per_point (empty until row 16 loads samples), --water-positions mirrored into water_position first; oods status LakeExportSpec: a loaded area's export decodes with BeachSnapshot.decode and BeachFinder reads it from MAROLA_BEACHES_DIR without a request; a fixture sample's export round-trips through CachedWaterQualityClient and carries its water position; a second export is byte-identical 9
11 beach-workflow marola-oods — beach-etl.yml per §5.4: Monday cron, dispatch with area and dry_run, concurrency: oods-lake, the catalog download (an empty listing starts a new catalog, which DuckLakeStore migrates on attach from the pinned lake contract; a refused one stops) / oods beaches / maintain / upload (always) / export steps, the catalog steps as a composite action in .github/actions/oods-lake/, CLOUDFLARE_R2_* mapped to OODS_*; marola-image bumped to the first image with marola.oods.Main; README, AGENTS.md and docs/3-development.md say the workflows write the bucket, never this repo, and name their secrets and variables actionlint. A dry_run dispatch on the branch, run by a person, is green and writes nothing (run link). After merge, a person dispatches area=all twice: the first creates the catalog at the contract's version, the second prints changed=0 snapshot=- (run links, rows, catalog size and seconds in the PR) 8, 10, 22
12 planner marola-app — Planner (pure): partitions per source, state and year; the mutability rule of §5.4; immutable partitions with a matching hash in knownPartitions skipped; an unknown UF is an error PlannerSpec: mode × date × known-partition tables, including the January rule at 2027-01-10 and its absence at 2027-02-20; an unknown UF fails 3
13 throttle marola-app — Throttle: ≥ 250 ms per host, ≤ 4 in flight, 3 attempts with backoff on 5xx and time-outs, stop on 429/403 ThrottleSpec with a fake clock and a recording Http.Transport: spacing per host; a 500 then 200 succeeds in two attempts; three 500s fail; a 429 stops before the next request 1
14 ima-sc-adapter marola-app — ImaScAdapter: incremental from /relatorio/mapa (reusing ImaScWaterQualityClient.parse's field handling), backfill from the CSV per beach-year, points from the feed; e_coli, NMP/100mL, <20 → (20, below); fixtures already in oods/src/test/resources/ima-sc/ ImaScAdapterSpec: MIP-0056.tasks row 2's cases (2003: 31 rows, 2010: 29, 2025: 145, 2026: 90; the censored value; both empty shapes; every Campeche row resolves to one of five UUIDs; an invented point name keeps a slug key with geo_source = none) 12, 13
15 sources-positions marola-oods — etl/sources.json (ima-sc only: institute, state, level, channel, cron, brazil_only) and etl/water-positions.csv (header only), both checked by etl-inputs-check.sh etl-inputs-check.sh --self-test adds: a source with an unknown channel fails; a position row with one of three fields set fails; one outside Brazil's box fails 8
16 water-load marola-app — Load and oods load: plan → fetch through Throttle → adapter rows → upsertPartition with its fetch_partition row → fetch_run; --dry-run; --max-minutes → partial; the source table mirrored from --sources; per-source status line; exit codes LoadSpec on a local lake: a second run changes no row and makes no snapshot; a 5xx → failed, rolled back; a 429 stops without a retry; a backfill killed after two partitions resumes with the third 5, 14
17 water-workflow marola-oods — water-quality-etl.yml per §5.4, SC only: the plan job, the load matrix with max-parallel: 1, concurrency: oods-lake, row 11's oods-lake composite action, timeout-minutes: 330; marola-image bumped to an image with row 16 actionlint. A dry_run dispatch for SC, run by a person, is green (run link). After merge, a person dispatches the SC backfill, then the weekly run twice: requests, partitions, MB and minutes in the PR, the second run no_new_bulletin 11, 15, 16
18 inea-rj-adapter marola-app — IneaRjAdapter: city pages → zones from the last 45 days → PDFs → IneaPdfParser; curated coordinates; uncurated codes kept with geo_source = 'none'; enterococci; the filename date IneaRjAdapterSpec from captured pages and a zone PDF to exact rows, an uncurated code kept, not dropped 16
19 inema-ba-adapter marola-app — InemaBaAdapter: the newest campaign (not the pinned 83453), InemaPdfParser, curated coordinates; no sample date presented as a lab date InemaBaAdapterSpec from a captured campaign list and PDF to exact rows; the campaign picked is the newest in the fixture 16
20 rj-ba-workflow marola-oods — inea-rj and inema-ba in etl/sources.json (brazil_only: true), the Friday and Saturday crons, scripts/br-proxy.sh from marola-dev/marola-site#20 started before a brazil_only state, MAROLA_BR_PROXY passed only then; marola-image bumped actionlint; etl-inputs-check.sh green. After merge and once the proxy exists, a person dispatches RJ and BA (run links) 17, 18, 19
21 mip-implemented marola — MIP-0075 flipped to Implemented with every PR number mip_graph.py --check; docs-lint green 6, 7, 11, 17, 20, 22
22 lake-schema marola-oods — schema first: lake/migrations/0001_init.sql (§5.2's nine tables, sample partitioned by source_id, and schema_migration(version, name, checksum, applied_at)), lake/views.sql and lake/checks.sql (moved from specs/001-beach-persistence/contracts/); scripts/lake-migrate.sh applying them to a local lake only (just lake-migrate, run by just quality): each migration one transaction with its row and md5, a duplicate version or an edited applied migration refused, views.sql's md5 as the version-0 row; scripts/lake-contract.sh and a v*-tag release.yml attaching marola-oods-lake-<tag>.tar.gz; the DuckDB CLI pinned in flake.nix. Nothing here writes the bucket. On top of marola-dev/marola-oods#3 lake-migrate.sh --self-test: an empty local lake gets every table, the partitioning and schema_migration = 1 with both md5s; views.sql and checks.sql run on it; a second run applies nothing and makes no snapshot; a broken migration rolls back and records nothing; a duplicate version and an edited migration are refused; a changed views.sql is applied once. lake-contract.sh --self-test: the tarball is byte-identical after a touch and a chmod –

The people's gates

Not rows: no agent can do them, and none blocks writing or reviewing a row's code.

  • Before row 4: marola-dev/marola-oods#22 merged, and a person tags marola-oods v0.1.0 so release.yml attaches the first marola-oods-lake-v0.1.0.tar.gz.
  • Before row 11's first real dispatch: §11's Phase 2 exception; §5.6's bucket, ETL token, marola-oods secrets and smoke test.
  • Before row 20's dispatch: the Brazil proxy VM and MAROLA_BR_PROXY, with its cost confirmed (marola-dev/marola-site#20).
  • Every dispatch of a workflow is a person's (marola-oods AGENTS.md).

Decisions

  1. This list replaces MIP-0056.tasks rows 1–8 for the store. MIP-0056 wrote raw files and Parquet into git on marola-oods's main; this MIP keeps its adapters, planner, columns and fixtures and writes the lake instead (§5.1). Row 14's tests are MIP-0056.tasks row 2's. MIP-0056's PDF channel (its row 7) stays a later row of its own.
  2. The site's read path is not here. site.yml's download of exports/ with the read-only token (§5.5) is marola-site's own task, filed there once row 11's exports exist.
  3. Facility and trail snapshot readers (so a build makes no Overpass call at all) are a follow-up in marola-app after row 10, not part of creating the tables.
  4. Rows 1–11 and 22 are tracked in marola-oods (the maintainer, 2026-10-06), whichever repo the PR lands in, under the parent marola-dev/marola#672; a marola-app PR closes its row with Closes marola-dev/marola-oods#N.
  5. Schema first (the maintainer, 2026-10-05; revised 2026-10-06 on marola-dev/marola-oods#22): the tables are numbered migrations in marola-oods's lake/, next to the views and checks, released by tag as marola-oods-lake-<tag>.tar.gz. marola-app pins the tag (row 4) and DuckLakeStore applies only what is pending, on attach inside an oods-lake job (row 5), so the bucket's catalog is created by the first beach-etl.yml run and never written from a laptop. The schema has one home, the SQL; a later change is a new NNNN_*.sql, never a rebuilt lake. The Scala row types are checked against it (row 4), not generated from it: kyo-schema 1.0.0-RC7 has no DDL generator and kyo-sql no DuckDB backend. Row 22 is listed last to keep the filed issue numbers; it runs first.
  6. Optional later: a backfill that re-dispatches itself with GITHUB_TOKEN, capped in hops.