Cases

INFRA-440 · Infra Ops

orders.fulfillment consumer lag

SEV-3 · Waiting

  1. 1

    Signal

  2. 2

    Cause

  3. 3

    Plan

  4. 4

    Draft

  5. 5

    Person gate

  6. 6

    Record

OpenTelemetry

Consumer lag on orders.fulfillment crossed the rubric. The topic is still accepting writes.

Cause

fulfill-worker stalled after a deploy. The broker and orders-db are fine.

Plan

Restart the worker. Do not rebalance the topic.

Draft

Restart fulfill-worker

Allow-list · restart-deployment · Jira INFRA-440

Footprint

CustomersChannelsAPIsServicesAsyncDataHostsRetail customersCustomer · 3Partner channelCustomer · 1storefrontChannel · 2checkoutChannel · 3mobile-appChannel · 4orders-apiAPI · 9payments-apiAPI · 5auth-apiAPI · 1pricing-apiAPI · 3catalog-apiAPI · 2inventory-apiAPI · 3orders.fulfillmentKafka · 4payments.eventsKafka · 2fulfill-workerWorker · 3orders-dbDatabase · 2payments-dbDatabase · 1catalog-dbDatabase · 2card-networkExternal · 1app-node-aHost · 2app-node-cHost · 1

Person gate

The write waits here.

Restart fulfill-worker is on the allow-list (restart-deployment). It does not run until you decide.

  • JiraDry-run
  • OpenTelemetryDry-run
  • KubernetesDry-run