Dropbox open sources pb-jelly, a protobuf code generation framework for Rust

0

by

pb-jelly is a protobuf code generation framework for the Rust language developed at Dropbox.

Historical previous

This implementation turn into once first and most well-known written in 2016 to fulfill the need of shuffling honorable quantity
of bytes in Dropbox’s Storage System (Magic Pocket).
Previously, we had been the exercise of rust-protobuf (and therefore generated APIs are precisely
the same to rating migration easy) nonetheless serializing Rust structs to proto messages, and then serializing them again in
our RPC layer, supposed a few copies (and same thing in reverse on parsing stack). Taking administration of this
implementation and integrating it in our RPC stack quit-to-quit helped steer clear of these additional copies.

Over time, the implementation has grown and matured and is currently pale in a few aspects of Dropbox, including
our Sync Engine, and the aforementioned Magic Pocket.

Varied implementations exist in the Rust ecosystem (e.g. prost and rust-protobuf), we desired to part ours as successfully.

Crates.io Documentation Crates.io

Aspects

  • Helpful “Rust-minded” proto extensions, e.g. [(rust.box_it)=true]
  • Scalable – Generates separate crates per module, with possibility for crate-per-directory
    • Autogenerates Cargo.toml, or optionally Spec.toml / bazel BUILD files
  • Make stronger for Serde
  • Zero-replica deserialization with Bytes by arrangement of a proto extension [(rust.zero_copy)=true]
  • Automatically bins messages if it finds a recursive message definition
  • Retains feedback on proto fields
  • Helps proto2 and proto3


Extensions

Extension Description Form Instance
(rust.zero_copy)=lawful Generates discipline form of Indolent for proto bytes fields to enhance zero-replica deserialization Self-discipline zero_copy
(rust.box_it)=lawful Generates a Field discipline form Self-discipline box_it
(rust.form)="form" Generates a custom discipline form Self-discipline custom_type
(rust.preserve_unrecognized)=lawful Preserves unrecognized proto fields into an _unrecognized struct discipline Self-discipline TODO
(gogoproto.nullable)=false Generates non-nullable fields kinds Self-discipline TODO
(rust.nullable)=false Generates oneofs as non-nullable (fail on deserialization) Oneof non_optional
(rust.err_if_default_or_unknown)=lawful Generates enums as non-zeroable (fail on deserialization) Enum non_optional
(rust.serde_derive)=lawful Generates serde serializable/deserializable messages File serde


Utilizing pb-jelly in your project

Quite so much of crates, a few languages, my oh my!

A must-rating Crates

There are finest two crates you may perhaps well need whenever you must rating to exercise this with you project pb-jelly and pb-jelly-gen.

pb-jelly

Contains the overall major traits and structs that vitality our generated code, e.g. Message and Indolent. Embody this as a dependency, e.g.

[dependencies]
pb-jelly = "0.0.1"
pb-jelly-gen

A framework for generating Rust structs and implementations for proto2 and proto3 files. Embody this as a produce-dependency, e.g.

[build-dependencies]
pb-jelly-gen = "0.0.1"

Generating Rust Code

In yell to generate Rust code from your proto definitions you may perhaps well need three issues

  1. pb-jelly-gen
  2. protoc – The protobuf compiler, this can even be built from provide protobuf or installed by arrangement of brew set up protobuf.
  3. python2 – The codegen plugin pale with protoc is written in Python2. Sooner than working it, which you may perhaps per chance rating to set up some programs, a necessities.txt is pending #18.

Snatch a look on the examples crate to circulate attempting how we leverage pb-jelly-gen and produce.rs to web began the exercise of protobufs in Rust!


Non-major Crates

  • pb-take a look at incorporates integration assessments and benchmarks. You rating no longer rating to apprehension about this one unless you must rating to make contributions to this repository!
  • pb-kinds incorporates generated Rust kinds for successfully identified proto kinds [TODO]: Could deprecate this?


A Show masks On Scalability 📝

We reward “scalabilty” as a feature, what does that mean? We employ an opinionated stance that every module wants to be a crate, as against generating Rust files 1:1 with proto files. We employ this stance because rustc is parallel accurate by arrangement of crates, nonetheless no longer yet totally parallel interior a crate. After we had all of our generated Rust code in a single crate, it turn into once customarily that single crate that took the longest to compile. The resolution to these long compile times, turn into once constructing many crates!


First, contributions are vastly most traditional and highly encouraged. For lawful reasons all originate air
contributors must comply with Dropbox’s CLA. Thank you for
your conception.



A pair of of the aspects here require extra tooling to be reliable, which are no longer yet public.

  • Spec.toml is a stripped down templated Cargo.toml – which which you may perhaps per chance also script convert into
    Cargo.toml in yell to web constant dependency versions in a multi-crate project.
    For the time being, the script to convert Spec.toml -> Cargo.toml is never any longer yet on hand
  • Autogenerated BUILD files require extra tooling to convert BUILD.in-gen-proto~ to a BUILD file

Closed structs with public fields

  • Adding fields to a proto file will result in compiler errors. This in most cases is a revenue in that it permits the
    compiler to establish all callsites that may perhaps well ought to be visited. Alternatively, it’ll rating updating protos with
    many callsites a diminutive bit gradual. We opted to circulate this path to rating it more uncomplicated to add a brand unique discipline and replace
    all callsites with the support of the compiler.

Carrier Skills

  • Generating stubs for gPRC possibilities and servers
  1. Clone Repo.
  2. Set up Dependencies / Checking out Dependencies. These instructions are for OSX the exercise of the brew
    equipment supervisor. Advise the correct equipment supervisor for your system.

    • protoc – fragment of Google’s protobuf tools
      • brew set up protobuf
    • Set up Lunge
      • brew set up coast
    • Set up gogoproto
      • coast web github.com/gogo/protobuf/proto
    • Set up Python & dependencies
      • brew set up python3
      • pip set up six
      • pip set up protobuf
    • Generate take a look at protos
      • On OSX: which you may perhaps per chance rating to set up coreutils for realpath brew set up coreutils
      • ./gen_protos.sh
  3. pb-jelly currently makes exercise of an experimental take a look at framework that requires a nightly produce of rust.
    • rustup default nightly
  4. cargo take a look at

TODO

  • Doc and add assessments for grpc_slices
  • Add custom_type instance to pbtest
  • Add travis-ci integration
  • Mypy the codegen.py in CI
  • rustfmt in CI
  • Add provider generation codegen as successfully
  • Rename pbtest.proto to pbtest2.proto
  • Carry out github disorders for closing todos
  • Speed mypy-protobuf for the assessments – to larger mypy codegen.py
  • Autogenerate warn on rust 2018 idioms
  • Add take a look at files w/ a few generated crates that depend upon every other
  • Decide out how to host documentation somewhere
  • Bonus: Port over the take a look at which serializes in coast and deserializes/reserializes in rust
  • Bonus: Python3
  • Originate provide oxidize (or one thing of the form)

Contributors

Dropboxers

The same Projects

rust-protobuf – Rust implementation of Google protocol buffers

prost – PROST! a Protocol Buffers implementation for the Rust Language

like a flash-protobuf – A rust implementation of protobuf parser

serde-protobuf

Read More

Leave A Reply

Your email address will not be published.