Most of what I test at work happens after the API has already said yes. A handheld scans some RFID tags, a service accepts the request, and then the real work happens somewhere else: an event goes onto a Kafka topic, consumers pick it up, files land in S3, rows change in a database. If the test stops at the HTTP response, it has checked the easiest part.
Here’s the shape of a scenario that checks the rest, simplified:
Scenario: A store receives a shipment
Given a shipment of 3 cartons is dispatched to a test store
When the store handheld scans the cartons
Then a receiving event for the shipment is published
And the event's disposition is "in_progress"
And the S3 export contains the same tags as the event
And nothing for the shipment reaches the dead-letter queueA few things make a check like this trustworthy.
Find your event, not an event
A busy topic has plenty of messages that look like yours. The test needs a key it controls, such as a shipment ID it created itself, and it should only ever assert on messages carrying that key. Otherwise a stale message from someone else’s run can make a broken build pass.
Wait with a deadline, not a sleep
Asynchronous means the event arrives eventually. A fixed sleep is either too short, and the test is flaky, or too long, and the suite is slow. Polling until the event appears or a deadline passes gives you both speed and a clear failure message: “no receiving event for this shipment within 30 seconds” is a bug report, not a mystery.
Check the content, then where it went
An event arriving is not the same as an event being right. Assert on the fields that carry the business meaning: the business step, the disposition, the timestamp compared with when the scan happened. Then follow it downstream and compare what landed in S3 or the database with what was on the topic.
Check the place failures hide
When a consumer can’t process a message, it often goes to a dead-letter queue, and the happy path keeps looking fine. Asserting that the DLQ holds nothing for your key turns a silent failure into a red test. Testing the reverse matters too: send a message the consumer should reject, and check that it does land in the DLQ.
When it fails, the evidence is already there
The same steps that make the test reliable make it debuggable. Because each assertion is tied to one hop, a red step tells you where to look: the topic in Lenses, the service logs in Splunk, the data with SQL. That’s the difference between “the end-to-end test failed” and “the consumer dropped the event because the timestamp was in the wrong zone”.