What is Avro & Schema Registry?

Definition
Apache Avro is a binary, schema based serialization format that encodes data compactly while preserving strong types. A Schema Registry is a centralized service that stores schema versions, tracks evolution history, and enforces compatibility rules to prevent breaking changes.
The Fundamental Problem: Imagine hundreds of microservices and data pipelines exchanging messages over years. A producer changes a field from user_email to email_address in its JSON output. Older consumers reading from a Kafka topic suddenly fail because they expect the old field name. You cannot coordinate a simultaneous upgrade across all systems. This is the long term data compatibility problem.

Avro solves this by separating schema from data. Every record is encoded according to a writer schema. Readers decode using their own reader schema. As long as the two schemas are compatible, reading succeeds. Avro defines clear rules: adding a field with a default value is safe (backward compatible), but changing a field type from string to integer breaks compatibility.

Schema Registry as Coordinator: The registry stores every schema version under a subject (typically one per Kafka topic). When producers write messages, they embed a tiny schema identifier (just 5 bytes) instead of the full schema. Consumers use this identifier to fetch the correct schema from the registry, then Avro handles the writer to reader mapping.

Payload Size Comparison
1 KB
JSON
400 B
AVRO

This registry prevents breaking changes from being deployed. If a producer tries to register an incompatible schema, the registry rejects it. This acts as a contract enforcement layer, similar to how an API gateway governs REST endpoints.

💡 Key Takeaways

✓Avro encodes data in binary format with schemas defined separately, reducing payload size by 30 to 70 percent compared to JSON

✓Schema Registry acts as a centralized contract enforcement layer, storing all schema versions and preventing incompatible changes

✓Producers embed only a 5 byte schema identifier in messages, not the full schema, keeping overhead minimal

✓Consumers fetch writer schemas from the registry and use Avro resolution to map fields to their own reader schema

✓Compatibility rules (backward, forward, full) define what changes are safe, such as adding fields with defaults versus changing field types

📌 Examples

1A retail CDC pipeline ingesting 50,000 events per second uses Avro to reduce JSON payloads from 1 KB to 400 bytes, cutting network bandwidth from 400 Mbps to 160 Mbps

2A producer registers a new schema version adding an optional <code>phone_number</code> field with a default null value. The registry accepts it as backward compatible because old consumers can safely ignore the new field.

← Back to Avro & Schema Registry Overview