I created this table CREATE TABLE k_events WITH (KAFKA_TOPIC='t_events', KEY_FORMAT='KAFKA', VALUE_FORMAT='JSON') as SELECT id, COLLECT_LIST(output) as machine_condition from k_merged WINDOW TUMBLING (SIZE 5 SECOND) GROUP BY id emit changes; When I browse the topic, I can see the id as the key of the message. But, I don’t see in the message itself. For example, I am expecting the messages in t_events as { "id":"uuid", "machine_condition": [] } Instead, I only see { "machine_condition": []} What am I missing? 2 posts - 2 participants Read full topic ..read more

I created a stream in KSQLDB with data type as BIGINT. I am inserting some rows into this stream with hard coded values e.g., -1, 100, etc. It seems, the INSERT considers these hard coded values as INT and therefore, the INSERT fails. The error message is Incompatible schema between results and sink.. I can see that, the resulting schema from the INSERT statement shows INT whereas, the source schema i.e. the stream schema as BIGINT (as intended). Therefore, is there a way to BIGINT(-1) in the INSERT statement? Or, what is the guidance to insert hard coded integer values into a BIGINT column? 1 ..read more

Is there a KSQL way to merge i.e. concatenate multiple streams into one? For example, is there a KSQL way to achieve the same result as this tutorial? 4 posts - 2 participants Read full topic ..read more

“@type”:“statement_error”, “error_code”:40001, “message”:“Cannot cast io.confluent.ksql.parser.tree.StructAll to io.confluent.ksql.parser.tree.SingleColumn” I got the above error when trying to run a pull query that involves exploding a struct. The same command when run as a push query instead has no issue. I look at the ksqldb docs but it makes no mention that the ->* is valid for push queries only. 3 posts - 2 participants Read full topic ..read more

I did read about the Java client. My goal is to stream the output of the push queries via a HTTP SSE endpoint. This example seems to write the topic and not stream the query output. Laziness aside, I was wondering if you have come across such a sample project. 1 post - 1 participant Read full topic ..read more

This command fails: CREATE STREAM test WITH (KAFKA_TOPIC='test', PARTITIONS=10, VALUE_FORMAT='JSON_SR'); with Schema registry fetch for topic value request failed for topic: test. Whereas, in the schema registry, I can see a schema with test-value as subject. What am I missing? 4 posts - 2 participants Read full topic ..read more

I am using confluentinc/cp-ksqldb-server:7.6.1 and wish to reconfigure the logging so that a Logstash-compatible JSON output is produced. I tried using io.confluent.common.logging.log4j.StructuredJsonLayout, but unfortunately, it is not Logstash compatible out of the box (e.g. has a time field instead of the expected @timestamp) and is not configurable (to e.g. rename the fields). I’ve also tried to put e.g. log4j-layout-template-json:2.17.1 on the classpath and configuring the layout in log4j.properties, but am getting the following error: 2024-07-12T15:26:39.168891619Z log4j:ERROR A "org.ap ..read more

When creating a stream or a table using an existing topic, how do I tell ksql to create key column using Kafka message key? 2 posts - 2 participants Read full topic ..read more

Hi, I have nested arrays in my Kafka topic Array1 [{ String1 : "Value1", String2 : "Value2" Array2: [{ String1: "Value1", Int1: "Value"]}] What is the best way to Explode this, Nested Explodes doesn’t seem to work, appreciate any pointers, thanks. 1 post - 1 participant Read full topic ..read more

Trying the follow Want to count sales per terminal, window by hour, once got this working can add more fields like store name also… CREATE TABLE json_sales_per_terminal_point WITH (KAFKA_TOPIC='json_sales_per_terminal_point', VALUE_FORMAT='Json', PARTITIONS=1) as SELECT store->id as store_id, TerminalPoint as terminal_point, count(1) as sales_per_terminal FROM json_salescompleted1 WINDOW TUMBLING (SIZE 1 HOUR) group by store->id , TerminalPoint EMIT CHANGES; Example record { "InvoiceNumber": "1341243123341232", "SaleDate ..read more