Streaming output (STO) pattern

A pattern that processes data and delivers it to consumers in kafka topics

diagram

sto

sto pattern storytelling

  • Product owner defines dataproduct in catalog/metadata repos

  • Developers clone archetypes/patterns, implement code and push code to git

  • DevOps pipeline creates resources, tags,lifecycle deletion/retention policies, efc

  • CI/CD pipeline deploys code into pipeline data processors

  • Business user request tokens for secure access

  • IAM workflow creates policies

  • End user or end system requests execution schedule

  • Approval or auto/approval workflow schedules/launches execution pipelines

  • Kafka event notifies execution status

  • End user or app requests metadata

  • End user or app access data via kafka endpoint