Fix outdated ingestion pipeline docs

.spelling

@@ -117,6 +117,7 @@ DNS
 DNSSEC
 docs.telemetry.mozilla.org
 Dockerfile
+Dockerflow
 dropdown
 DSRE
 DTMO

src/concepts/pipeline/gcp_data_pipeline.md

@@ -16,13 +16,13 @@ graph TD
 
 f1(fa:fa-firefox Firefox) -->|HTTP Post| d0(fa:fa-filter Ingestion Edge)
 d0 --> p1(fa:fa-stream Raw Topic)
-p1 --> d1(fa:fa-exchange-alt Landfill Sink)
-d1 --> b1(fa:fa-database Landfill BQ)
+p1 --> d1(fa:fa-exchange-alt Raw Sink)
+d1 --> b1(fa:fa-database Raw BQ)
 p1 --> d2(fa:fa-exchange-alt Decoder)
 d2 -->|success| p2(fa:fa-stream Decoded Topic)
 d2 -.->|fail| p3(fa:fa-stream Errors Topic)
 p3 --> d4(fa:fa-exchange-alt Errors Sink)
-p2 --> d3(fa:fa-exchange-alt BigQuery Sink)
+p2 --> d3(fa:fa-exchange-alt Live Sink)
 d3 --> b2(fa:fa-database Live Tables BQ)
 d4 --> b3(fa:fa-database Error Tables BQ)
 
@@ -76,8 +76,10 @@ message body of optionally-gzipped JSON.
 These messages are forwarded to a PubSub message queue with minimal processing,
 and made available in a **Raw** topic.
 
-A [Dataflow] job reads this topic and writes the raw messages to a BigQuery **Landfill** sink.
-This Landfill data is not used for analysis, but is stored in its raw form for
+A Kubernetes `Raw Sink` job (part of the [`ingestion-sink`][ingestion-sink]
+service) reads this topic and writes the raw messages to a BigQuery
+`payload_bytes_raw` sink (historically called _Landfill_).
+This raw data is not used for analysis, but is stored in its raw form for
 recovery and backfill purposes.
 
 If there is a processing error or data-loss downstream in the pipeline, this is an important fail-safe.
@@ -129,11 +131,13 @@ and the [schema generator].
 The result are tables that contains SQL-friendly field names for all known
 measures, as implemented in the [probe scraper].
 
-A [Dataflow] job reads from the Decoded topic and writes out to
+A Kubernetes `Live Sink` job (part of the [`ingestion-sink`][ingestion-sink]
+service) reads from the Decoded topic and writes out to
 **[live ping tables][table layout]**.
 These tables are updated frequently, and typically reflect data within a few
-minutes of it being ingested. They are optimized for accessing recent data,
-but are only guaranteed to contain a few days of history.
+minutes of it being ingested. They are intended for low-latency access to
+recent data and are not the supported interface for analysis; for that, use the
+**[historical ping tables][table layout]** (stable tables) described below.
 
 Historical raw ping data is stored in **[historical ping tables][table layout]**,
 also known as **stable tables**.
@@ -234,4 +238,5 @@ build and work with aggregate datasets rather than individual-level data.
 [norm]: ../channels/channel_normalization.md
 [bqe]: https://github.com/mozilla/bigquery-etl
 [gcp-ingestion]: https://github.com/mozilla/gcp-ingestion
+[ingestion-sink]: https://github.com/mozilla/gcp-ingestion/tree/main/ingestion-sink
 [looker]: https://mozilla.cloud.looker.com

src/concepts/pipeline/http_edge_spec.md

@@ -23,14 +23,23 @@ documents described by a [JSONSchema].
 
 ### GET Request
 
-Accept GET on `/status`, returning `OK` if all is well. This can be used to
-check the health of web servers.
+The edge server does not serve application data over GET. It exposes
+[Dockerflow] health-check endpoints:
+
+- `/__lbheartbeat__` - returns 200 unconditionally; used by the load balancer
+  and Kubernetes probes to confirm the server is up.
+- `/__heartbeat__` - returns 200 when internal checks pass (enough free disk
+  for the local queue, and the queue is accessible) and 500 otherwise.
+- `/__version__` - returns build and version information.
+
+[dockerflow]: https://github.com/mozilla-services/Dockerflow
 
 ### GET Response codes
 
-- _200_ - OK. `/status` and all's well
-- _404_ - Any GET other than `/status`
-- _500_ - All is not well
+- _200_ - a health check passed (`/__lbheartbeat__`, or `/__heartbeat__` when
+  all checks pass), or version info was returned (`/__version__`)
+- _404_ - any other GET path
+- _500_ - a `/__heartbeat__` check failed
 
 ### POST/PUT Request
 
@@ -67,19 +76,20 @@ A specific example:
 Here the `namespace` is fixed as "telemetry", and there is no `docVersion` in the URL.
 This means that incoming JSON documents must be parsed to determine the schema version
 to apply for validation. This logic is part of the downstream [decoder] job.
-Also note the required query parameter suffix `?v=4`.
-Documents sent under `/submit/telemetry` without `v=4` will be rejected at the edge.
+Also note the query parameter suffix `?v=4`, a historical convention sent by
+Firefox Desktop. The edge server routes on the URL path only and forwards the
+request body opaquely; the query string is preserved as request metadata.
 
 ### POST/PUT Response codes
 
 - _200_ - OK. Request accepted into the pipeline.
 - _400_ - Bad request, for example an un-encoded space in the URL.
 - _404_ - not found - POST/PUT to an unknown namespace
 - _405_ - wrong request type (anything other than POST/PUT)
-- _411_ - missing content-length header
-- _413_ - request body too large (Note that if we have badly-behaved clients that retry on `4XX`, we may opt to send back 202 on body/path too long).
-- _414_ - request path too long (See above)
+- _413_ - request body too large
+- _431_ - a request header (or other captured metadata attribute) exceeds PubSub's 1024 byte attribute limit
 - _500_ - internal error
+- _507_ - insufficient storage - the edge's local retry queue is full (for example during an extended PubSub outage)
 
 ### Supported HTTP Headers
 
@@ -132,3 +142,5 @@ is restored and messages can be flushed to the queue.
 Based on [past outages], this is typically a few hours or less.
 
 [past outages]: https://status.cloud.google.com/incident/cloud-pubsub
+[schemas repository]: https://github.com/mozilla-services/mozilla-pipeline-schemas
+[decoder]: gcp_data_pipeline.md#decoding

src/concepts/pipeline/schemas.md

@@ -236,10 +236,11 @@ LIMIT 1
 
 #### Data Structure Normalization
 
-Thee decoder is also responsible for transforming the data to
-accommodate BigQuery limitations in data representation. All transformations are defined in
-[`ingestion-beam` under
-`com.mozilla.telemtry.transforms.PubsubMessageToTableRow`](https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-beam/src/main/java/com/mozilla/telemetry/transforms/PubsubMessageToTableRow.java).
+The decoder is also responsible for transforming the data to
+accommodate BigQuery limitations in data representation. Most transformations are implemented in
+[`ingestion-core` under
+`com.mozilla.telemetry.ingestion.core.transform.PubsubMessageToObjectNode`](https://github.com/mozilla/gcp-ingestion/blob/main/ingestion-core/src/main/java/com/mozilla/telemetry/ingestion/core/transform/PubsubMessageToObjectNode.java);
+`ingestion-beam`'s `PubsubMessageToTableRow` is a thin wrapper that adds Beam metrics and converts the result to a BigQuery `TableRow`.
 
 The following transformations are currently applied: