How can we put a variant message ( one of a few message types ) inside a protobuf message?

Implementing Variant Messages in Protocol Buffers with oneof

Learn how to efficiently define and use variant message types within Protocol Buffers using the oneof keyword, enabling flexible and type-safe communication.

In distributed systems and microservices, it's common for a single communication channel or API endpoint to handle multiple, distinct message types. For instance, a notification service might send email, SMS, or push notifications, each with its own specific data structure. Protocol Buffers (Protobuf) provides a powerful feature called oneof that elegantly solves this problem, allowing you to define a field that can hold one of a set of possible message types at any given time.

Understanding the oneof Keyword

The oneof keyword in Protobuf allows you to define a field that can be any one of several types. This is particularly useful when you have a message that needs to carry different kinds of data depending on the context, but only one of those data types will be present at any given moment. It's a more efficient and type-safe alternative to using optional fields and checking which ones are set, or defining a base message and extending it (which Protobuf doesn't directly support in the same way as object-oriented inheritance).

flowchart TD
    A[Sender] --> B{Message Type?}
    B -->|Email| C[EmailNotification]
    B -->|SMS| D[SMSNotification]
    B -->|Push| E[PushNotification]
    C --> F(NotificationWrapper `oneof` field)
    D --> F
    E --> F
    F --> G[Receiver]
    style F fill:#f9f,stroke:#333,stroke-width:2px

Conceptual flow of different notification types being encapsulated by a oneof field.

Defining oneof in a .proto File

To use oneof, you declare it within your message definition, followed by a block containing the fields that can be part of that oneof group. Each field within the oneof block must have a unique field number. When you set one field in a oneof group, all other fields in that same oneof group are automatically cleared. This ensures that only one variant is ever present.

syntax = "proto3";

package notifications;

message EmailNotification {
  string recipient_email = 1;
  string subject = 2;
  string body = 3;
}

message SMSNotification {
  string phone_number = 1;
  string message_content = 2;
}

message PushNotification {
  string device_token = 1;
  string title = 2;
  string alert_body = 3;
  map<string, string> custom_data = 4;
}

message NotificationWrapper {
  string notification_id = 1;
  int64 timestamp = 2;

  oneof notification_type {
    EmailNotification email_notification = 3;
    SMSNotification sms_notification = 4;
    PushNotification push_notification = 5;
  }
}

Example .proto definition using oneof for different notification types.

Working with oneof in Generated Code

When you compile your .proto file, Protobuf generates code in your chosen language that provides methods to interact with the oneof field. Typically, there will be a method to check which field within the oneof is currently set (e.g., has_email_notification() or getNotificationTypeCase()), and methods to set or get the individual fields. This allows for type-safe handling of the variant message.

Java

NotificationWrapper wrapper = NotificationWrapper.newBuilder() .setNotificationId("noti-123") .setTimestamp(System.currentTimeMillis()) .setEmailNotification(EmailNotification.newBuilder() .setRecipientEmail("user@example.com") .setSubject("Hello") .setBody("This is an email.") .build()) .build();

// Check which type is set switch (wrapper.getNotificationTypeCase()) { case EMAIL_NOTIFICATION: System.out.println("Email Subject: " + wrapper.getEmailNotification().getSubject()); break; case SMS_NOTIFICATION: System.out.println("SMS Content: " + wrapper.getSmsNotification().getMessageContent()); break; case PUSH_NOTIFICATION: System.out.println("Push Title: " + wrapper.getPushNotification().getTitle()); break; case NOTIFICATIONTYPE_NOT_SET: System.out.println("No notification type set."); break; }

Python

from notifications_pb2 import NotificationWrapper, EmailNotification, SMSNotification, PushNotification

wrapper = NotificationWrapper( notification_id="noti-456", timestamp=1678886400000, email_notification=EmailNotification( recipient_email="another@example.com", subject="Greetings", body="This is another email." ) )

Check which type is set

if wrapper.HasField('email_notification'): print(f"Email Subject: {wrapper.email_notification.subject}") elif wrapper.HasField('sms_notification'): print(f"SMS Content: {wrapper.sms_notification.message_content}") elif wrapper.HasField('push_notification'): print(f"Push Title: {wrapper.push_notification.title}") else: print("No notification type set.")

Go

package main

import ( "fmt" "time" pb "path/to/your/notifications" )

func main() { emailNoti := &pb.EmailNotification{ RecipientEmail: "go@example.com", Subject: "Go Lang", Body: "Hello from Go!", }

wrapper := &pb.NotificationWrapper{
	NotificationId: "noti-789",
	Timestamp:      time.Now().UnixMilli(),
	NotificationType: &pb.NotificationWrapper_EmailNotification{
		EmailNotification: emailNoti,
	},
}

// Check which type is set
switch x := wrapper.GetNotificationType().(type) {
case *pb.NotificationWrapper_EmailNotification:
	fmt.Printf("Email Subject: %s\n", x.EmailNotification.GetSubject())
case *pb.NotificationWrapper_SmsNotification:
	fmt.Printf("SMS Content: %s\n", x.SmsNotification.GetMessageContent())
case *pb.NotificationWrapper_PushNotification:
	fmt.Printf("Push Title: %s\n", x.PushNotification.GetTitle())
case nil:
	fmt.Println("No notification type set.")
default:
	fmt.Printf("Unknown notification type: %T\n", x)
}

}

Benefits and Use Cases of oneof

The oneof feature offers several advantages:

• Space Efficiency: Only the data for the currently set field is serialized, reducing message size compared to having many optional fields where only one is used.
• Type Safety: It enforces that only one variant can be present, preventing logical errors where multiple conflicting data types might be set simultaneously.
• Clarity: The .proto definition clearly indicates that these fields are mutually exclusive, improving readability and understanding of the message structure.
• Flexibility: It allows for evolving message structures without breaking compatibility, as new variant types can be added to a oneof group.

Common use cases include:

• Command/Event Messages: A single Command message might contain CreateUserCommand, UpdateProductCommand, or DeleteOrderCommand.
• API Responses: A generic Response message could hold SuccessResponse, ErrorResponse, or InProgressResponse.
• Configuration: A ConfigValue message might be an IntValue, StringValue, BooleanValue, etc.
• Polymorphic Data: Representing different kinds of data that share a common wrapper, like the notification example.