Skip to content

Latest commit

 

History

History
111 lines (72 loc) · 4.25 KB

concepts.md

File metadata and controls

111 lines (72 loc) · 4.25 KB

Concepts

Overview Quick Start Concepts Syntax Reference Demo Examples FAQ Roadmap

Table of Contents

Components

The main components of KSQL are CLI, engine, and the REST interface.

CLI

Provides a familiar interface, designed users of MySQL, Postgres, etc.

Engine

Performs the data processing.

REST interface

Enables an engine to receive instructions from the CLI.

Terminology

When using KSQL, the following terminology is used.

Stream

A stream is an unbounded sequence of structured data ("facts"). For example, we could have a stream of financial transactions such as "Alice sent $100 to Bob, then Charlie sent $50 to Bob". Facts in a stream are immutable, which means new facts can be inserted to a stream, but existing facts can never be updated or deleted. Streams can be created from a Kafka topic or derived from existing streams and tables.

Table

A table is a view of a stream, or another table, and represents a collection of evolving facts. For example, we could have a table that contains the latest financial information such as "Bob’s current account balance is $150". It is the equivalent of a traditional database table but enriched by streaming semantics such as windowing. Facts in a table are mutable, which means new facts can be inserted to the table, and existing facts can be updated or deleted. Tables can be created from a Kafka topic or derived from existing streams and tables.

Modes of operation

Standalone mode

In stand-alone mode, both the KSQL client and server components are co-located on the same machine, in the same JVM, and are started together which makes it convenient for local development and testing.

Standalone mode

Here's an overview of running KSQL in standalone mode:

  • Starts a CLI, an Engine, and a REST server all in the same JVM
  • Ideal for laptop development

    • Use with default settings:

      ./bin/ksql-cli local

    • Use with custom settings:

      ./bin/ksql-cli local --properties-file foo/bar/ksql.properties

Client-server mode

In client-server mode, you can run a pool of KSQL servers on remote machines, VMs, or containers and the CLI connects to them over HTTP.

Client-server mode

Here's an overview of running KSQL in client-server mode:

  • Start any number of server nodes

    • Use with default settings:

      ./bin/ksql-server-start

    • Use with custom settings:

      ./bin/ksql-server-start --properties-file foo.properties

  • Start any number of CLIs, specifying a server address as remote endpoint

    ./bin/ksql-cli remote http://server:8090

  • All engines share the work, for example, instances of the same KSQL apps. You can scale up or down without restarting.