-
-
Notifications
You must be signed in to change notification settings - Fork 403
Produce raw message
- 1. Introduction
- 2. What is RAW message
- 3. How to produce a RAW message to a Kafka topic
- 4. Which SerDe is used for this kind of record
- 5. How the SerDe is configured for a Kafka producer
- 6. How does the full "produce" operation step looks like
- 7. How to run the test
- 8. How to verify the record in the topic via kafka-console-consumer
- 9. Where is the docker file
In this Wiki page, we will see what is a RAW message and how to produce this kind of records to a Kafka topic.
For more details about Kafka testing, please visit Kafka Testing Introduction - Baby Steps
RAW messages can be any kind of messages which may or may not have any defined structure, as well as the messages, may or may not conform to a specific data schema. Think of a RAW message as plain text for now.
We just need to specify the "recordType" as RAW while producing the record to a Kafka topic.
"recordType" : "RAW"
And the operation here is
"operation" : "produce"
The test request will look like below
"request": {
"recordType": "RAW",
"records": [
{
"key": "KEY-101",
"value": "Hello World"
}
]
}
Please observe above, the "value" field which is a simple text message.
"value": "Hello World"
You can put anything into it and send to Kafka topic and Kafka broker will accept it.
e.g. "value": "bla bla" or "value": "10478ABCDG1234-HEXA-VALUE" etc
Then the consumer needs to decide how it should consume it, the best way to consume this kind of message is same as RAW. See the consumer example Consume RAW message in the side bar.
What is SerDe? Ans: Serializer, Deserializer.
The best way to produce and consume RAW messages is use,
- StringSerializer
- key.serializer=org.apache.kafka.common.serialization.StringSerializer
- value.serializer=org.apache.kafka.common.serialization.StringSerializer
- StringDeserializer
- key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
- value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
- See kafka_test_server.properties here how they are configured.
- Look for these two configs in the same resource folder.
- kafka.producer.properties=kafka_servers/kafka_producer.properties
- kafka.consumer.properties=kafka_servers/kafka_consumer.properties
You can update/add/remove the desired key-value depending on your test-requirement.
Now we have all the details to invoke the "produce" operation. The step looks like,
test_kafka_produce_raw.json
---------------------------
{
"name": "produce_raw_msg",
"url": "kafka-topic:demo-raw-topic-1",
"operation": "produce",
"request": {
"recordType": "RAW",
"records": [
{
"key": "KEY-101",
"value": "Hello World"
}
]
},
"assertions": {
"status": "Ok",
"recordMetadata": "$NOT.NULL"
}
}
The test looks like below and we can run the test as usual JUnit test,
@TargetEnv("kafka_servers/kafka_test_server.properties")
@RunWith(ZeroCodeUnitRunner.class)
public class KafkaProduceTest {
@Test
@JsonTestCase("kafka/produce/test_kafka_produce_raw.json")
public void testProduce_raw() throws Exception {
}
}
Where, the "kafka_test_server.properties" file contains,
kafka.bootstrap.servers=localhost:9092
kafka.producer.properties=kafka_servers/kafka_producer.properties
kafka.consumer.properties=kafka_servers/kafka_consumer.properties
And the "kafka_producer.properties" contains,
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
# kafka producer properties
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
client.id=zerocode-producer
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
You can use many variation of "Serializer" depending upon your "key" and "value" data types of your tests. e.g. A "DoubleSerializer" if the Key is a Double or Integer or Long value.
key.serializer=org.apache.kafka.common.serialization.DoubleSerializer
$ docker exec compose_kafka_1 kafka-console-consumer --bootstrap-server localhost:29092 --topic demo-ksql --from-beginning
Hello World
Where, "compose_kafka_1" is the name of the Kafka server container.
All Docker files are here in zerocode-docker-factory, but to run this test, you can use "kafka-single-node.yml" or "kafka-schema-registry.yml".
See intructions in the side bar on how to run the "compose" files. In short,
To bring the containers up,
docker-compose -f kafka-single-node.yml up -d
To bring the containers down,
docker-compose -f kafka-single-node.yml down
To view the kafka server logs
To bring the containers down, docker logs <kafka_container_name>
Visit the Zerocode Documentation Site for all things.
-
User's Guide
-
Matchers
-
Zerocode Value Tokens
-
YAML DSL
-
Http Testing
-
Kafka Testing
- Introduction
- Produce, consume proto message
- Produce raw message
- Consume raw message
- Produce JSON message
- Consume JSON message
- Produce and consume XML message
- Kafka - consume the latest message or n latest messages
- Produce avro message
- Consume avro message
- KSQL in action
- Produce multiple records
- Produce from file
- Produce to a partition
- Produce and consume records with headers
- Produce n assert partition ack
- Comsume and dump to file
- commitSync vs commitAsync
- Overriding config inside a test
- Chosing String or Int or Avro Serializer
- Chosing String or Int or Avro Deserializer
- Attaching timestamp during load
- Default timestamp provided by Kafka
- Consume and assert avro schema metadata
- Error handling - produce via avro schema
- Sorting Kafka records consumed
-
DB Testing
-
Kotlin Testing
-
Performance Testing - Load and Stress
- Performance Testing - via awesome JUnit runners
- Load Vs Stress generation on target application
- Run a single test or a scenario in parallel
- Run multiple test scenarios in parallel - Production load simulation
- Dynamically change the payload for every request
- Analytics - Useful report(s) or statistics
-
Parameterized Testing
-
Docker
-
More+
-
Extensions
-
JUnit5 Jupiter Test
-
Questions And Answers(FAQ)
- What is Zerocode testing?
- SSL http https connections supported?
- How to assert array size Greater-Than Lesser-Than etc?
- How to invoke POST api?
- How to assert custom headers of the response?
- How to pass custom security token into the request header?
- When to use JUnit Suite runner and when Zerocode Package runner?
- How to execute DB SQL and assert?
- How to handle Http response other than utf-8 e.g. utf-16 or utf-32 ?
- Random Number Generator Placeholders Usages and Limits
- Automation tests for Zerocode lib itself
- Picking a leaf value from the array matching JSON Path
- Array assertions made easy, incl. size and element finder
-
Read Our Blogs
- Top 16 Open Source API Testing Tools For REST & SOAP Services - joecolantonio (Lists popular tools - Globally)
- OAuth2 Test Automation - DZone 2min Read
- Zero defect APIs - Build Pipe Line - Medium 10 min Read
- Develop ZeroDefect API's with ZeroCode! - Extreme Portal ( A must read for all developers and test engineers) 10min Read
- Performance testing using JUnit and maven - Codeproject 10 min Read
- REST API or SOAP End Point Testing - Codeproject 10min Read
- DZone- MuleSoft API Testing With Zerocode Test Framework - DZone 5min Read
- Testing need not be harder or slower, it should be easier and faster - DZone 5 min Read
- Kotlin Integration Testing simplified via Zerocode - Extreme portal 10 min Read
- and More...