Monday Mini: Protobufs from JSON in Java

On my pet project, I am using protocol buffers as Interface Language (IDL). Using a simple, shared definition should speed things up, right? Well, kind of!

Protocol Buffers are a message format commonly used in combination with Remote Procedure Calls (RPC; or specifically, GRPC if you’re doing RPC with protobufs). If you’re working in a microservices world, you can’t really just assume every service uses these protocol buffered messages (I’ll just call them protobufs from now on) over an RPC styled interface. Heck, for plain CRUD use cases that don’t see a lot of throughput, JSON over REST is probably just fine. Then again, the resources we disclose through the REST interface, could very well map to a protobuf type or message.

Consider; a gaming system with, for today’s example, a ‘unit’ datatype; with stats like name, model, point value, armor points, special skills, etc. In the protobuf IDL, it would look a bit like this:

message UnitStats {
  string name=1;
  string model=2;
  int32 pointvalue=3;
  ...
  repeated string specials=14;
}

The cool thing about protocol buffers is that there is plenty of tooling around to generate code from something like the code above; I could easily integrate tooling into my build cycle to generate Java or .NET classes, Go structs, or even Javascript code for frontends. Saves me the trouble of writing the same code over and over. Nice.

This gaming system would have a library service, holding all kinds of statistics for every possible unit imagine thousands of possible units we could use. Possibly in a fancy document database or something. Using JSON and REST as an interface is the best fit for this use case - no hassle, the default operations we want with GET/PUT/POST/DELETE, very little implementation overhead.

But then, once the game is actually underway, ‘instances’ of these units, up to hundreds a game, are loaded onto a board, and get passed around between all kinds of other services. Then, the ‘overhead’ of JSON, as well as the more RPC like nature of game methods and phases, units as actors, and so on, kind of makes us want to move to something like protobuf to keep things scalable.

Yes, yes, it’s a stretch. I could write the game more efficiently in just one big GM (Glorious Monolith). And it would work a lot faster too. And if I used Go it would compile faster. Run faster too. Require less resources.

But!

It’d be less fun.

So bear with me.

The set-up

How to deal with this? I was writing a bit of logic to start a game, which, for the sake of technology, means a ‘gamemaster’ in Go, invokes a (protobuf) method on a ‘armybuilder, build in Java, to create an army. That little service would then use REST on a ‘library’ to get some unit data. The library is Go again. Sorry. No love for C# yet. So sue me. (Please actually don’t and be patient. Sheesh)

So, somewhere in that ‘armybuilder’ we’ll need to map from the JSON from our rest resource to protobuf. Which means in the first instance, I had to unmarshal the json response. Either into a custom class, or use some generic JSONElement classes eg;

    var jsonObject = new JsonParser().parse(new String(response)).getAsJsonObject();
    var unitBuilder = UnitStats.newBuilder();

    unitBuilder.setName(jsonObject.get("name").getAsString());
    unitBuilder.setModel(jsonObject.get("model").getAsString());
    unitBuilder.setPointvalue(jsonObject.get("pointvalue").getAsInt());
    if (jsonObject.has("role")) {
      unitBuilder.setRole(jsonObject.get("role").getAsString());
    }
    // and a About 10 more properties

This is silly; as both the library and the armybuilder use generated code from the very same protobuf definition on a what a ‘unit’ is. And if things can be null in the JSON, as shown, we’ll need some way of checking that. Yes, yes, the above example is with the gson library, and I’m sure there are better ways of handling with some other library that supports Optionals, but still!

The play

After some googling, I found that the package com.google.protobuf:protobuf-java-util has a very convenient JsonFormat class that we can use to merge applicable values from a json string into a protobuf (builder) object.

Now, instead of all of the above, we get:

      var unit = CommonUnitProtos.UnitStats.newBuilder();
      byte[] response = client
          .invokeMethod("library", "units/by/random", "", HttpExtension.GET, null, byte[].class).block(); // Don't mind the DAPR
      parser.merge(new String(response), unit);
      result.addUnits(unit);

For our use case this works pretty flawlessly. We know both the Go library REST service, as well as the above java armybuilder code use the same protobuf definitions. They both know what a unit is. The JsonFormat parsers just saves us the hassle of writing unnecessary boilerplate code.

Of course, it would have been even more clean if we could tell Dapr that instead of wanting a byte[].class, we could just pass in our protobuf type. But unfortunately, if Dapr receives a JSON instead of the expected protobuf, that doesn’t seem to work. Admittedly, documentation in the Java API of Dapr leaves to be desired too, so, there’s that as well. I’m sure things will get more clear in the future.

A lot less error-prone this way still. You love to see it.

The Evaluation

Maybe I’m just making too big a a fuzz out of this; but I’m just kind of happy that this allows me, for my project, to have a different kind of GM. Not a glorious monolith, but rather, a glorious monorepo. With one definition of protobuf types and messages I can reuse across various microservices, written in the language I want to fiddle around with.

The above ‘result’ can be rather easily replicated in Go as well, as the generated Go structs are annotated with ‘json’ tags. For C# there seems to be a Google.Protobuf.JsonParser, which even seems to be included in the default protobuf package; So, luckily, it doesn’t seem to be that weird to want the things I’m wanting.

In conclusion, to me, it’s a bit a mystery why, for Java, the above util class is not in included in the standard java protocol buffer package. And because it isn’t, you can’t just depend on the Dapr java client and get this nifty utility

Who to blame? Google for keeping keeping their core protobuf package so light? Lightweight makes sense! Everybody should do that!! Microsoft for not including the reference in their Java client SDK? I mean, no, it makes sense not to include something you don’t use.

There you have it, just some perfect sensical, but unfortunate turn of events that made me dedicate these 1000+ words on a Monday night. Hope you enjoyed.