🪴 Anil's Garden

Python Generated Code Guide Protocol Buffers Documentation

12 Dec 202422 min read

Python Generated Code Guide | Protocol Buffers Documentation

Excerpt

Describes exactly what Python definitions the protocol buffer compiler generates for any given protocol definition.

Describes exactly what Python definitions the protocol buffer compiler generates for any given protocol definition.

Any differences between proto2 and proto3 generated code are highlighted - note that these differences are in the generated code as described in this document, not the base message classes/interfaces, which are the same in both versions. You should read the proto2 language guide and/or proto3 language guide before reading this document.

The Python Protocol Buffers implementation is a little different from C++ and Java. In Python, the compiler only outputs code to build descriptors for the generated classes, and a Python metaclass does the real work. This document describes what you get after the metaclass has been applied.

Compiler Invocation

The protocol buffer compiler produces Python output when invoked with the --python_out= command-line flag. The parameter to the --python_out= option is the directory where you want the compiler to write your Python output. The compiler creates a .py file for each .proto file input. The names of the output files are computed by taking the name of the .proto file and making two changes:

  • The extension (.proto) is replaced with _pb2.py.
  • The proto path (specified with the --proto_path= or -I command-line flag) is replaced with the output path (specified with the --python_out= flag).

So, for example, let’s say you invoke the compiler as follows:

protoc --proto_path=src --python_out=build/gen src/foo.proto src/bar/baz.proto

The compiler will read the files src/foo.proto and src/bar/baz.proto and produce two output files: build/gen/foo_pb2.py and build/gen/bar/baz_pb2.py. The compiler will automatically create the directory build/gen/bar if necessary, but it will not create build or build/gen; they must already exist.

Protoc can generate Python stubs (.pyi) using the --pyi_out parameter.

Note that if the .proto file or its path contains any characters which cannot be used in Python module names (for example, hyphens), they will be replaced with underscores. So, the file foo-bar.proto becomes the Python file foo_bar_pb2.py.

Packages

The Python code generated by the protocol buffer compiler is completely unaffected by the package name defined in the .proto file. Instead, Python packages are identified by directory structure.

Messages

Given a simple message declaration:

The protocol buffer compiler generates a class called Foo, which subclasses google.protobuf.Message. The class is a concrete class; no abstract methods are left unimplemented. Unlike C++ and Java, Python generated code is unaffected by the optimize_for option in the .proto file; in effect, all Python code is optimized for code size.

If the message’s name is a Python keyword, then its class will only be accessible via getattr(), as described in the Names which conflict with Python keywords section.

You should not create your own Foo subclasses. Generated classes are not designed for subclassing and may lead to “fragile base class” problems. Besides, implementation inheritance is bad design.

Python message classes have no particular public members other than those defined by the Message interface and those generated for nested fields, messages, and enum types (described below). Message provides methods you can use to check, manipulate, read, or write the entire message, including parsing from and serializing to binary strings. In addition to these methods, the Foo class defines the following static methods:

  • FromString(s): Returns a new message instance deserialized from the given string.

Note that you can also use the text_format module to work with protocol messages in text format: for example, the Merge() method lets you merge an ASCII representation of a message into an existing message.

Nested Types

A message can be declared inside another message. For example:

message Foo { message Bar {} }

In this case, the Bar class is declared as a static member of Foo, so you can refer to it as Foo.Bar.

Well Known Types

Protocol buffers provides a number of well-known types that you can use in your .proto files along with your own message types. Some WKT messages have special methods in addition to the usual protocol buffer message methods, as they subclass both google.protobuf.Message and a WKT class.

Any

For Any messages, you can call Pack() to pack a specified message into the current Any message, or Unpack() to unpack the current Any message into a specified message. For example:

any_message.Pack(message) any_message.Unpack(message)

Unpack() also checks the descriptor of the passed-in message object against the stored one and returns False if they don’t match and does not attempt any unpacking; True otherwise.

You can also call the Is() method to check if the Any message represents the given protocol buffer type. For example:

assert any_message.Is(message.DESCRIPTOR)

Use the TypeName() method to retrieve the protobuf type name of an inner message.

Timestamp

Timestamp messages can be converted to/from RFC 3339 date string format (JSON string) using the ToJsonString()/FromJsonString() methods. For example:

timestamp_message.FromJsonString("1970-01-01T00:00:00Z") assert timestamp_message.ToJsonString() == "1970-01-01T00:00:00Z"

You can also call GetCurrentTime() to fill the Timestamp message with current time:

timestamp_message.GetCurrentTime()

To convert between other time units since epoch, you can call ToNanoseconds(), FromNanoseconds(), ToMicroseconds(), FromMicroseconds(), ToMilliseconds(), FromMilliseconds(), ToSeconds(), or FromSeconds(). The generated code also has ToDatetime() and FromDatetime() methods to convert between Python datetime objects and Timestamps. For example:

timestamp_message.FromMicroseconds(-1) assert timestamp_message.ToMicroseconds() == -1 dt = datetime(2016, 1, 1) timestamp_message.FromDatetime(dt) self.assertEqual(dt, timestamp_message.ToDatetime())

Duration

Duration messages have the same methods as Timestamp to convert between JSON string and other time units. To convert between timedelta and Duration, you can call ToTimedelta() or FromTimedelta. For example:

duration_message.FromNanoseconds(1999999999) td = duration_message.ToTimedelta() assert td.seconds == 1 assert td.microseconds == 999999

FieldMask

FieldMask messages can be converted to/from JSON string using the ToJsonString()/FromJsonString() methods. In addition, a FieldMask message has the following methods:

  • IsValidForDescriptor(message_descriptor): Checks whether the FieldMask is valid for Message Descriptor.
  • AllFieldsFromDescriptor(message_descriptor): Gets all direct fields of Message Descriptor to FieldMask.
  • CanonicalFormFromMask(mask): Converts a FieldMask to the canonical form.
  • Union(mask1, mask2): Merges two FieldMasks into this FieldMask.
  • Intersect(mask1, mask2): Intersects two FieldMasks into this FieldMask.
  • MergeMessage(source, destination, replace_message_field=False, replace_repeated_field=False): Merges fields specified in FieldMask from source to destination.

Struct

Struct messages let you get and set the items directly. For example:

struct_message["key1"] = 5 struct_message["key2"] = "abc" struct_message["key3"] = True

To get or create a list/struct, you can call get_or_create_list()/get_or_create_struct(). For example:

struct.get_or_create_struct("key4")["subkey"] = 11.0 struct.get_or_create_list("key5")

ListValue

A ListValue message acts like a Python sequence that lets you do the following:

list_value = struct_message.get_or_create_list("key") list_value.extend([6, "seven", True, None]) list_value.append(False) assert len(list_value) == 5 assert list_value[0] == 6 assert list_value[1] == "seven" assert list_value[2] == True assert list_value[3] == None assert list_Value[4] == False

To add a ListValue/Struct, call add_list()/add_struct(). For example:

list_value.add_struct()["key"] = 1 list_value.add_list().extend([1, "two", True])

Fields

For each field in a message type, the corresponding class has a property with the same name as the field. How you can manipulate the property depends on its type.

As well as a property, the compiler generates an integer constant for each field containing its field number. The constant name is the field name converted to upper-case followed by _FIELD_NUMBER. For example, given the field optional int32 foo_bar = 5;, the compiler will generate the constant FOO_BAR_FIELD_NUMBER = 5.

If the field’s name is a Python keyword, then its property will only be accessible via getattr() and setattr(), as described in the Names which conflict with Python keywords section.

Singular Fields (proto2)

If you have a singular (optional or required) field foo of any non-message type, you can manipulate the field foo as if it were a regular field. For example, if foo’s type is int32, you can say:

message.foo = 123 print(message.foo)

Note that setting foo to a value of the wrong type will raise a TypeError.

If foo is read when it is not set, its value is the default value for that field. To check if foo is set, or to clear the value of foo, you must call the HasField() or ClearField() methods of the Message interface. For example:

assert not message.HasField("foo") message.foo = 123 assert message.HasField("foo") message.ClearField("foo") assert not message.HasField("foo")

Singular Fields (proto3)

If you have a singular field foo of any non-message type, you can manipulate the field foo as if it were a regular field. For example, if foo’s type is int32, you can say:

message.foo = 123 print(message.foo)

Note that setting foo to a value of the wrong type will raise a TypeError.

If foo is read when it is not set, its value is the default value for that field. To clear the value of foo and reset it to the default value for its type, you call the ClearField() method of the Message interface. For example:

message.foo = 123 message.ClearField("foo")

Singular Message Fields

Message types work slightly differently. You cannot assign a value to an embedded message field. Instead, assigning a value to any field within the child message implies setting the message field in the parent. You can also use the parent message’s HasField() method to check if a message type field value has been set.

So, for example, let’s say you have the following .proto definition:

message Foo { optional Bar bar = 1; } message Bar { optional int32 i = 1; }

You cannot do the following:

foo = Foo() foo.bar = Bar() # WRONG!

Instead, to set bar, you simply assign a value directly to a field within bar, and - presto! - foo has a bar field:

foo = Foo() assert not foo.HasField("bar") foo.bar.i = 1 assert foo.HasField("bar") assert foo.bar.i == 1 foo.ClearField("bar") assert not foo.HasField("bar") assert foo.bar.i == 0 # Default value

Similarly, you can set bar using the Message interface’s CopyFrom() method. This copies all the values from another message of the same type as bar.

Note that simply reading a field inside bar does not set the field:

foo = Foo() assert not foo.HasField("bar") print(foo.bar.i) # Print i's default value assert not foo.HasField("bar")

If you need the “has” bit on a message that does not have any fields you can or want to set, you may use the SetInParent() method.

foo = Foo() assert not foo.HasField("bar") foo.bar.SetInParent() # Set Foo.bar to a default Bar message assert foo.HasField("bar")

Repeated Fields

Repeated fields are represented as an object that acts like a Python sequence. As with embedded messages, you cannot assign the field directly, but you can manipulate it. For example, given this message definition:

message Foo { repeated int32 nums = 1; }

You can do the following:

foo = Foo() foo.nums.append(15) # Appends one value foo.nums.extend([32, 47]) # Appends an entire list assert len(foo.nums) == 3 assert foo.nums[0] == 15 assert foo.nums[1] == 32 assert foo.nums == [15, 32, 47] foo.nums[:] = [33, 48] # Assigns an entire list assert foo.nums == [33, 48] foo.nums[1] = 56 # Reassigns a value assert foo.nums[1] == 56 for i in foo.nums: # Loops and print print(i) del foo.nums[:] # Clears list (works just like in a Python list)

The ClearField() method of the Message interface works in addition to using Python del.

When using the index to retrieve a value, you can use negative numbers, such as using -1 to retrieve the last element in the list. If your index goes out of bounds, you’ll get an IndexError: list index out of range.

Repeated Message Fields

Repeated message fields work similar to repeated scalar fields. However, the corresponding Python object also has an add() method that creates a new message object, appends it to the list, and returns it for the caller to fill in. Also, the object’s append() method makes a copy of the given message and appends that copy to the list. This is done so that messages are always owned by the parent message to avoid circular references and other confusion that can happen when a mutable data structure has multiple owners. Similarly, the object’s extend() method appends an entire list of messages, but makes a copy of every message in the list.

For example, given this message definition:

message Foo { repeated Bar bars = 1; } message Bar { optional int32 i = 1; optional int32 j = 2; }

You can do the following:

foo = Foo() bar = foo.bars.add() # Adds a Bar then modify bar.i = 15 foo.bars.add().i = 32 # Adds and modify at the same time new_bar = Bar() new_bar.i = 40 another_bar = Bar() another_bar.i = 57 foo.bars.append(new_bar) # Uses append() to copy foo.bars.extend([another_bar]) # Uses extend() to copy assert len(foo.bars) == 4 assert foo.bars[0].i == 15 assert foo.bars[1].i == 32 assert foo.bars[2].i == 40 assert foo.bars[2] == new_bar # The appended message is equal, assert foo.bars[2] is not new_bar # but it is a copy! assert foo.bars[3].i == 57 assert foo.bars[3] == another_bar # The extended message is equal, assert foo.bars[3] is not another_bar # but it is a copy! foo.bars[1].i = 56 # Modifies a single element assert foo.bars[1].i == 56 for bar in foo.bars: # Loops and print print(bar.i) del foo.bars[:] # Clears list # add() also forwards keyword arguments to the concrete class. # For example, you can do: foo.bars.add(i=12, j=13) # Initializers forward keyword arguments to a concrete class too. # For example: foo = Foo( # Creates Foo bars=[ # with its field bars set to a list Bar(i=15, j=17), # where each list member is also initialized during creation. Bar(i=32), Bar(i=47, j=77), ] ) assert len(foo.bars) == 3 assert foo.bars[0].i == 15 assert foo.bars[0].j == 17 assert foo.bars[1].i == 32 assert foo.bars[2].i == 47 assert foo.bars[2].j == 77

Unlike repeated scalar fields, repeated message fields don’t support item assignment (i.e. __setitem__). For example:

foo = Foo() foo.bars.add(i=3) # WRONG! foo.bars[0] = Bar(i=15) # Raises an exception # WRONG! foo.bars[:] = [Bar(i=15), Bar(i=17)] # Also raises an exception # WRONG! # AttributeError: Cannot delete field attribute del foo.bars # RIGHT del foo.bars[:] foo.bars.extend([Bar(i=15), Bar(i=17)])

Groups (proto2)

Note that groups are deprecated and should not be used when creating new message types – use nested message types instead.

A group combines a nested message type and a field into a single declaration, and uses a different wire format for the message. The generated message has the same name as the group. The generated field’s name is the lowercased name of the group.

For example, except for wire format, the following two message definitions are equivalent:

// Version 1: Using groups message SearchResponse { repeated group SearchResult = 1 { optional string url = 1; } } // Version 2: Not using groups message SearchResponse { message SearchResult { optional string url = 1; } repeated SearchResult searchresult = 1; }

A group is either required, optional, or repeated. A required or optional group is manipulated using the same API as a regular singular message field. A repeated group is manipulated using the same API as a regular repeated message field.

For example, given the above SearchResponse definition, you can do the following:

resp = SearchResponse() resp.searchresult.add(url="https://blog.google") assert resp.searchresult[0].url == "https://blog.google" assert resp.searchresult[0] == SearchResponse.SearchResult(url="https://blog.google")

Map Fields

Given this message definition:

message MyMessage { map<int32, int32> mapfield = 1; }

The generated Python API for the map field is just like a Python dict:

# Assign value to map m.mapfield[5] = 10 # Read value from map m.mapfield[5] # Iterate over map keys for key in m.mapfield: print(key) print(m.mapfield[key]) # Test whether key is in map: if 5 in m.mapfield: print(“Found!”) # Delete key from map. del m.mapfield[key]

As with embedded message fields, messages cannot be directly assigned into a map value. Instead, to add a message as a map value you reference an undefined key, which constructs and returns a new submessage:

m.message_map[key].submessage_field = 10

You can find out more about undefined keys in the next section.

Referencing undefined keys

The semantics of Protocol Buffer maps behave slightly differently to Python dicts when it comes to undefined keys. In a regular Python dict, referencing an undefined key raises a KeyError exception:

>>> x = {} >>> x[5] Traceback (most recent call last): File "<stdin>", line 1, in <module> KeyError: 5

However, in Protocol Buffers maps, referencing an undefined key creates the key in the map with a zero/false/empty value. This behavior is more like the Python standard library defaultdict.

>>> dict(m.mapfield) {} >>> m.mapfield[5] 0 >>> dict(m.mapfield) {5: 0}

This behavior is especially convenient for maps with message type values, because you can directly update the fields of the returned message.

>>> m.message_map[5].foo = 3

Note that even if you don’t assign any values to message fields, the submessage is still created in the map:

>>> m.message_map[10] <test_pb2.M2 object at 0x7fb022af28c0> >>> dict(m.message_map) {10: <test_pb2.M2 object at 0x7fb022af28c0>}

This is different from regular embedded message fields, where the message itself is only created once you assign a value to one of its fields.

As it may not be immediately obvious to anyone reading your code that m.message_map[10] alone, for example, may create a submessage, we also provide a get_or_create() method that does the same thing but whose name makes the possible message creation more explicit:

# Equivalent to: # m.message_map[10] # but more explicit that the statement might be creating a new # empty message in the map. m.message_map.get_or_create(10)

Enumerations

In Python, enums are just integers. A set of integral constants are defined corresponding to the enum’s defined values. For example, given:

message Foo { enum SomeEnum { VALUE_A = 0; VALUE_B = 5; VALUE_C = 1234; } optional SomeEnum bar = 1; }

The constants VALUE_A, VALUE_B, and VALUE_C are defined with values 0, 5, and 1234, respectively. You can access SomeEnum if desired. If an enum is defined in the outer scope, the values are module constants; if it is defined within a message (like above), they become static members of that message class.

For example, you can access the values in the three following ways for the following enum in a proto:

enum SomeEnum { VALUE_A = 0; VALUE_B = 5; VALUE_C = 1234; }
value_a = myproto_pb2.SomeEnum.VALUE_A # or myproto_pb2.VALUE_A # or myproto_pb2.SomeEnum.Value('VALUE_A')

An enum field works just like a scalar field.

foo = Foo() foo.bar = Foo.VALUE_A assert foo.bar == 0 assert foo.bar == Foo.VALUE_A

If the enum’s name (or an enum value) is a Python keyword, then its object (or the enum value’s property) will only be accessible via getattr(), as described in the Names which conflict with Python keywords section.

The values you can set in an enum depend on your protocol buffers version:

  • In proto2, an enum cannot contain a numeric value other than those defined for the enum type. If you assign a value that is not in the enum, the generated code will throw an exception.
  • proto3 uses open enum semantics: enum fields can contain any int32 value.

Enums have a number of utility methods for getting field names from values and vice versa, lists of fields, and so on - these are defined in enum_type_wrapper.EnumTypeWrapper (the base class for generated enum classes). So, for example, if you have the following standalone enum in myproto.proto:

enum SomeEnum { VALUE_A = 0; VALUE_B = 5; VALUE_C = 1234; }

…you can do this:

self.assertEqual('VALUE_A', myproto_pb2.SomeEnum.Name(myproto_pb2.VALUE_A)) self.assertEqual(5, myproto_pb2.SomeEnum.Value('VALUE_B'))

For an enum declared within a protocol message, such as Foo above, the syntax is similar:

self.assertEqual('VALUE_A', myproto_pb2.Foo.SomeEnum.Name(myproto_pb2.Foo.VALUE_A)) self.assertEqual(5, myproto_pb2.Foo.SomeEnum.Value('VALUE_B'))

If multiple enum constants have the same value (aliases), the first constant defined is returned.

enum SomeEnum { option allow_alias = true; VALUE_A = 0; VALUE_B = 5; VALUE_C = 1234; VALUE_B_ALIAS = 5; }

In the above example, myproto_pb2.SomeEnum.Name(5) returns "VALUE_B".

Oneof

Given a message with a oneof:

message Foo { oneof test_oneof { string name = 1; int32 serial_number = 2; } }

The Python class corresponding to Foo will have properties called name and serial_number just like regular fields. However, unlike regular fields, at most one of the fields in a oneof can be set at a time, which is ensured by the runtime. For example:

message = Foo() message.name = "Bender" assert message.HasField("name") message.serial_number = 2716057 assert message.HasField("serial_number") assert not message.HasField("name")

The message class also has a WhichOneof method that lets you find out which field (if any) in the oneof has been set. This method returns the name of the field that is set, or None if nothing has been set:

assert message.WhichOneof("test_oneof") is None message.name = "Bender" assert message.WhichOneof("test_oneof") == "name"

HasField and ClearField also accept oneof names in addition to field names:

assert not message.HasField("test_oneof") message.name = "Bender" assert message.HasField("test_oneof") message.serial_number = 2716057 assert message.HasField("test_oneof") message.ClearField("test_oneof") assert not message.HasField("test_oneof") assert not message.HasField("serial_number")

Note that calling ClearField on a oneof just clears the currently set field.

Names which conflict with Python keywords

If the name of a message, field, enum, or enum value is a Python keyword, then the name of its corresponding class or property will be the same, but you’ll only be able to access it using Python’s getattr() and setattr() built-in functions, and not via Python’s normal attribute reference syntax (i.e. the dot operator).

For example, if you have the following .proto definition:

message Baz { optional int32 from = 1 repeated int32 in = 2; }

You would access those fields like this:

baz = Baz() setattr(baz, "from", 99) assert getattr(baz, "from") == 99 getattr(baz, "in").append(42) assert getattr(baz, "in") == [42]

By contrast, trying to use obj.attr syntax to access these fields results in Python raising syntax errors when parsing your code:

# WRONG! baz.in # SyntaxError: invalid syntax baz.from # SyntaxError: invalid syntax

Extensions (proto2 only)

Given a message with an extension range:

message Foo { extensions 100 to 199; }

The Python class corresponding to Foo will have a member called Extensions, which is a dictionary mapping extension identifiers to their current values.

Given an extension definition:

extend Foo { optional int32 bar = 123; }

The protocol buffer compiler generates an “extension identifier” called bar. The identifier acts as a key to the Extensions dictionary. The result of looking up a value in this dictionary is exactly the same as if you accessed a normal field of the same type. So, given the above example, you could do:

foo = Foo() foo.Extensions[proto_file_pb2.bar] = 2 assert foo.Extensions[proto_file_pb2.bar] == 2

Note that you need to specify the extension identifier constant, not just a string name: this is because it’s possible for multiple extensions with the same name to be specified in different scopes.

Analogous to normal fields, Extensions[...] returns a message object for singular messages and a sequence for repeated fields.

The Message interface’s HasField() and ClearField() methods do not work with extensions; you must use HasExtension() and ClearExtension() instead. To use the HasExtension() and ClearExtension() methods, pass in the field_descriptor for the extension you are checking for the existence of.

Services

If the .proto file contains the following line:

option py_generic_services = true;

Then the protocol buffer compiler will generate code based on the service definitions found in the file as described in this section. However, the generated code may be undesirable as it is not tied to any particular RPC system, and thus requires more levels of indirection that code tailored to one system. If you do NOT want this code to be generated, add this line to the file:

option py_generic_services = false;

If neither of the above lines are given, the option defaults to false, as generic services are deprecated. (Note that prior to 2.4.0, the option defaults to true)

RPC systems based on .proto-language service definitions should provide plugins to generate code appropriate for the system. These plugins are likely to require that abstract services are disabled, so that they can generate their own classes of the same names. Plugins are new in version 2.3.0 (January 2010).

The remainder of this section describes what the protocol buffer compiler generates when abstract services are enabled.

Interface

Given a service definition:

service Foo { rpc Bar(FooRequest) returns(FooResponse); }

The protocol buffer compiler will generate a class Foo to represent this service. Foo will have a method for each method defined in the service definition. In this case, the method Bar is defined as:

def Bar(self, rpc_controller, request, done)

The parameters are equivalent to the parameters of Service.CallMethod(), except that the method_descriptor argument is implied.

These generated methods are intended to be overridden by subclasses. The default implementations simply call controller.SetFailed() with an error message indicating that the method is unimplemented, then invoke the done callback. When implementing your own service, you must subclass this generated service and implement its methods as appropriate.

Foo subclasses the Service interface. The protocol buffer compiler automatically generates implementations of the methods of Service as follows:

  • GetDescriptor: Returns the service’s ServiceDescriptor.
  • CallMethod: Determines which method is being called based on the provided method descriptor and calls it directly.
  • GetRequestClass and GetResponseClass: Returns the class of the request or response of the correct type for the given method.

Stub

The protocol buffer compiler also generates a “stub” implementation of every service interface, which is used by clients wishing to send requests to servers implementing the service. For the Foo service (above), the stub implementation Foo_Stub will be defined.

Foo_Stub is a subclass of Foo. Its constructor takes an RpcChannel as a parameter. The stub then implements each of the service’s methods by calling the channel’s CallMethod() method.

The Protocol Buffer library does not include an RPC implementation. However, it includes all of the tools you need to hook up a generated service class to any arbitrary RPC implementation of your choice. You need only provide implementations of RpcChannel and RpcController.

Plugin Insertion Points

Code generator plugins which want to extend the output of the Python code generator may insert code of the following types using the given insertion point names.

  • imports: Import statements.
  • module_scope: Top-level declarations.

Sharing Messages Between Python and C++

Prior to the 4.21.0 version of the Protobuf Python API, Python apps could share messages with C++ using a native extension. Starting in the 4.21.0 API version, sharing messages between Python and C++ is not supported by the default install. To enable this capability when working with the 4.x and later versions of the Protobuf Python API, define the environment variable, PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp, and ensure that the Python/C++ extension is installed.

Graph View

Backlinks

  • No backlinks found