Cap'n Proto

<h2 id="technical-overview">Technical overview</h2>
<h3>IDL Schema</h3>
Like most <a href="/facts/Remote_Procedure_Call/uMMb9CQw">RPC</a> frameworks dating as far back as <a href="/facts/Sun_RPC/oSR4C7hH">Sun RPC</a> and <a href="/facts/DCE%2fRPC/7XQyL8Hf">OSF DCE RPC</a> (and their object-based descendants <a href="/facts/Common_Object_Request_Broker_Architecture/VUDlTHs5">CORBA</a> and <a href="/facts/Distributed_Component_Object_Model/mpDM1JXX">DCOM</a>), Cap'n Proto uses an <a href="/facts/Interface_description_language/FpByLf4X">Interface Description Language</a> (IDL) to generate RPC libraries in a variety of programming languages - automating many low level details such as handling network requests, converting between data types, etc. The Cap'n Proto interface schema uses a <a href="/facts/List_of_programming_languages_by_type/XuWNoawk">C-like syntax</a> and supports common <a href="/facts/Primitive_data_type/0c898228">primitives data types</a> (booleans, integers, floats, etc.), <a href="/facts/Composite_data_type/RBBfE1FY">compound types</a> (structs, lists, enums), as well as <a href="/facts/Generic_programming/kqtqJk7p">generics</a> and <a href="/facts/Dynamic_type/Tz0bAMS9">dynamic types</a>.<a class="footnote-ref" id="fnref:1" href="#fn:1">1</a> Cap'n Proto also supports object-oriented features such as multiple inheritance, which has been criticized for its complexity.<a class="footnote-ref" id="fnref:2" href="#fn:2">2</a> @0xa558ef006c0c123; # Unique identifiers are manually or automatically assigned to files and compound types

struct Date @0x5c5a558ef006c0c1 {
  year @0 :Int16; # @n marks order values were added to the schema
  month @1 :UInt8;
  day @2 :UInt8;
}

struct Contact @0xf032a54bcb3667e0 {
  name @0 :Text;
  birthday @2 :Date; # fields can be added anywhere in the definition, but their numbering must reflect the order in which they were added

phones @1 :List(PhoneNumber);

struct PhoneNumber { # Compound types without a static ID cannot be renamed, as automatic IDs are deterministically generated
    number @0 :Text;
    type @1 :PhoneType = mobile; # Default value

enum PhoneType {
 mobile @0;
 landline @1;
 }
 }
}
Values in Cap'n Proto messages are represented in <a href="/facts/Binary_code/v5YF0sQl">binary</a>, as opposed to text encoding used by "<a href="/facts/Human-readable_medium/GDE1fpHT">human-readable</a>" formats such as <a href="/facts/JSON/bs1rUAhd">JSON</a> or <a href="/facts/XML/clNgUfWk">XML</a>. Cap'n Proto tries to make the storage/network protocol appropriate as an in-memory format, so that no translation step is needed when reading data into memory or writing data out of memory.<a class="footnote-ref" id="fnref:3" href="#fn:3">3</a> For example, the representation of numbers (<a href="/facts/Endianness/wbrQnyu3">endianness</a>) was chosen to match the representation the most popular CPU architectures.<a class="footnote-ref" id="fnref:4" href="#fn:4">4</a> When the in-memory and wire-protocol representations match, Cap'n Proto can <a href="/facts/Zero-copy/cc13RBxU">avoid copying and encoding data</a> when creating or reading a message and instead <a href="/facts/Pointer_(computer_programming)/fPoNzy9b">point</a> to the location of the value in memory. Cap'n Proto also supports random access to data, meaning that any field can be read without having to read the entire message.<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a>
Unlike other binary serialization protocols such as <a href="/facts/Fast_Infoset/6T4kUXdd">XMI</a>, Cap'n Proto considers fine-grained <a href="/facts/Data_validation/tFCchJTM">data validation</a> at the RPC level an anti-feature that limits a protocol's ability to <a href="/facts/Schema_evolution/RfA2sV2y">evolve</a>. This was informed by experiences at Google where simply changing a field from mandatory to optional would cause complex operational failures.<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a><a class="footnote-ref" id="fnref:7" href="#fn:7">7</a> Cap'n Proto schemas are designed to be flexible as possible and pushes data validation to the application level, allowing arbitrary renaming of fields, adding new fields, and making concrete types generic.<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a> Cap'n Proto does, however, validate pointer bounds and type check individual values when they are first <a href="/facts/Mutator_method/sKfxYzFM">accessed</a>.<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a>
Enforcing complex schema constraints would also incur significant overhead,<a class="footnote-ref" id="fnref:10" href="#fn:10">10</a> negating the benefits of reusing in-memory data structures and preventing random access to data.<a class="footnote-ref" id="fnref:11" href="#fn:11">11</a> Cap'n Proto protocol is theoretically suitable<a class="footnote-ref" id="fnref:12" href="#fn:12">12</a> for very fast <a href="/facts/Inter-process_communication/cIQYyT6y">inter-process communication</a> (IPC) via immutable shared memory, but as of October 2020 none of the implementations support data passing via shared memory.<a class="footnote-ref" id="fnref:13" href="#fn:13">13</a> However, Cap'n Proto is still generally considered faster than Protocol Buffers and similar RPC libraries.<a class="footnote-ref" id="fnref:14" href="#fn:14">14</a><a class="footnote-ref" id="fnref:15" href="#fn:15">15</a>

<h3>Networking</h3>
Cap'n Proto RPC is network aware: supporting both handling of disconnects and <a href="/facts/Futures_and_promises/5Mpgxv4m">promise pipelining</a>, wherein a server pipes the output of one function into another function. This saves a client a round trip per successive call to the server without having to provide a dedicated API for every possible call graph. Cap'n Proto can be layered on top of TLS<a class="footnote-ref" id="fnref:16" href="#fn:16">16</a> and support for the <a href="/facts/Noise_Protocol_Framework/WAR8x10r">Noise Protocol Framework</a> is on the roadmap.<a class="footnote-ref" id="fnref:17" href="#fn:17">17</a> Cap'n Proto RPC is transport agnostic, with the mainline implementation supporting WebSockets, HTTP, TCP, and UDP.<a class="footnote-ref" id="fnref:18" href="#fn:18">18</a>

<h3>Capability security</h3>
The Cap'n Proto RPC standard has a rich capability security model based on the CapTP protocol used by the <a href="/facts/E_(programming_language)/XCLbIxm0">E programming language</a>.<a class="footnote-ref" id="fnref:19" href="#fn:19">19</a>

As of October 2020, the reference implementation only supports level 2.<a class="footnote-ref" id="fnref:20" href="#fn:20">20</a>

<h2 id="comparison-to-other-serialization-formats">Comparison to other serialization formats</h2>
Cap'n Proto is often compared to other zero-copy serialization formats, such as Google's <a href="/facts/FlatBuffers/QAzSy9wt">FlatBuffers</a> and Simple Binary Encoding (SBE).<a class="footnote-ref" id="fnref:21" href="#fn:21">21</a><a class="footnote-ref" id="fnref:22" href="#fn:22">22</a>

<h2 id="adoption">Adoption</h2>
Cap'n Proto was originally created for Sandstorm.io, a startup offering a web application hosting platform with capability-based security. After Sandstorm.io failed commercially, the development team was <a href="/facts/Acqui-hiring/ahIn8zSk">acqui-hired</a> by Cloudflare,<a class="footnote-ref" id="fnref:23" href="#fn:23">23</a> which uses Cap'n Proto internally.<a class="footnote-ref" id="fnref:24" href="#fn:24">24</a>

<h2 id="notes">Notes</h2>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Varda, Kenton. "Cap'n Proto Schema Language". Archived from the original on 2015-03-17. Retrieved 2020-09-05. <a href="https://capnproto.org/language.html" target="_blank">https://capnproto.org/language.html</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Denhardt, Ian (June 2019). "A Critique of the Cap'n Proto Schema Language". zenhack.net. Archived from the original on 2019-06-26. Retrieved 2020-10-10. <a href="https://zenhack.net/2019/06/25/a-critique-of-the-capnproto-schema-language.html" target="_blank">https://zenhack.net/2019/06/25/a-critique-of-the-capnproto-schema-language.html</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Unlike Apache Arrow, Cap'n Proto's in-memory values are not suited for sharing mutable data <a href="/wiki/Apache_Arrow" target="_blank">/wiki/Apache_Arrow</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Varda, Kenton. "Cap'n Proto: Introduction". Cap'n Proto Homepage. Archived from the original on 2015-03-17. Retrieved 2020-11-09. <a href="https://capnproto.org/index.html" target="_blank">https://capnproto.org/index.html</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Varda, Kenton. "Cap'n Proto: Encoding Spec". Cap'n Proto. Archived from the original on 2015-03-17. <a href="https://capnproto.org/encoding.html" target="_blank">https://capnproto.org/encoding.html</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Varda, Kenton. "FAQ § How do I make a field "required", like in Protocol Buffers?". Cap'n Proto. Archived from the original on 2015-03-18. Retrieved 2020-09-05. <a href="https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers" target="_blank">https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">Marking a field as required was removed from Protocol Buffers 3. <a href="https://developers.google.com/protocol-buffers/docs/proto3" target="_blank">https://developers.google.com/protocol-buffers/docs/proto3</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">"Cap'n Proto: Schema Language". capnproto.org. Retrieved 2020-10-10. <a href="https://capnproto.org/language.html#evolving-your-protocol" target="_blank">https://capnproto.org/language.html#evolving-your-protocol</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Varda, Kenton. "Cap'n Proto: Encoding Spec". Cap'n Proto. Archived from the original on 2015-03-17. <a href="https://capnproto.org/encoding.html" target="_blank">https://capnproto.org/encoding.html</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
<li id="fn:10">Assuming the data has already been allocated (e.g. in network buffers, read from disk) access becomes O(1). Additional serialization/deserialization steps (as required to inspect values) would limit performance to O(n). <a href="/wiki/Big_O_notation" target="_blank">/wiki/Big_O_notation</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></li>
<li id="fn:11">"Cap'n Proto: Cap'n Proto, FlatBuffers, and SBE". capnproto.org. Retrieved 2020-10-10. <a href="https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html" target="_blank">https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></li>
<li id="fn:12">Richardson, Corey (October 2016). "Robigalia: An Operating System for the Modern Era". robigalia.gitlab.io. Archived from the original on 2018-09-15. Retrieved 2020-10-10. <a href="https://robigalia.gitlab.io/book/rosme.html#_capabilities_in_phoma" target="_blank">https://robigalia.gitlab.io/book/rosme.html#_capabilities_in_phoma</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></li>
<li id="fn:13">Kenton, Varda (May 3, 2017). "Why is not intended that in-memory state be in Cap'n Proto / Protobuf objects?". Hacker News (news.ycombinator.com). Retrieved 2020-10-10. <a href="https://news.ycombinator.com/item?id=14252940" target="_blank">https://news.ycombinator.com/item?id=14252940</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></li>
<li id="fn:14">Naughton, Chris (Aug 24, 2018). "Protocol Benchmarks". Github. Archived from the original on 2018-08-30. Retrieved 2020-09-05. <a href="https://github.com/ChrisMacNaughton/proto_benchmarks" target="_blank">https://github.com/ChrisMacNaughton/proto_benchmarks</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></li>
<li id="fn:15">Parimi, Dinesh (2019). "Datacenter Tax Cuts: Improving WSC Efficiency Through Protocol Buffer Acceleration" (PDF). Archived (PDF) from the original on 2020-09-06. Retrieved 2020-09-05. <a href="https://people.eecs.berkeley.edu/~kubitron/courses/cs262a-F19/projects/reports/project4_report_ver2.pdf" target="_blank">https://people.eecs.berkeley.edu/~kubitron/courses/cs262a-F19/projects/reports/project4_report_ver2.pdf</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></li>
<li id="fn:16">"Cap'n Proto: Road Map". capnproto.org. Retrieved 2020-10-10. <a href="https://capnproto.org/roadmap.html" target="_blank">https://capnproto.org/roadmap.html</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></li>
<li id="fn:17">"Roadmap". Cap'n Proto. 2021-03-13. Archived from the original on 2015-03-17. <a href="https://capnproto.org/roadmap.html#rpc-protocol-features" target="_blank">https://capnproto.org/roadmap.html#rpc-protocol-features</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></li>
<li id="fn:18">"Cap'n Proto: C++ RPC". capnproto.org. Retrieved 2020-10-10. <a href="https://capnproto.org/cxxrpc.html" target="_blank">https://capnproto.org/cxxrpc.html</a> <a href="#fnref:18" class="footnote-back-ref">↩</a></li>
<li id="fn:19">"RPC Protocol". Cap'n Proto. Archived from the original on 2015-03-18. <a href="https://capnproto.org/rpc.html#protocol-features" target="_blank">https://capnproto.org/rpc.html#protocol-features</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></li>
<li id="fn:20">"Roadmap". Cap'n Proto. 2021-03-13. Archived from the original on 2015-03-17. <a href="https://capnproto.org/roadmap.html#rpc-protocol-features" target="_blank">https://capnproto.org/roadmap.html#rpc-protocol-features</a> <a href="#fnref:20" class="footnote-back-ref">↩</a></li>
<li id="fn:21">"Cap'n Proto: Cap'n Proto, FlatBuffers, and SBE". capnproto.org. Retrieved 2020-10-10. <a href="https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html" target="_blank">https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></li>
<li id="fn:22">"Why flatbuffers instead of capnp?". <a href="https://github.com/google/flatbuffers/issues/2" target="_blank">https://github.com/google/flatbuffers/issues/2</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></li>
<li id="fn:23">Varda, Kenton (13 Mar 2017). "The Sandstorm Team is joining Cloudflare". Sandstorm.io. Archived from the original on 2017-03-13. Retrieved 2020-09-05. <a href="https://sandstorm.io/news/2017-03-13-joining-cloudflare" target="_blank">https://sandstorm.io/news/2017-03-13-joining-cloudflare</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></li>
<li id="fn:24">Zhi, Jiale (2013). "Introducing lua-capnproto: better serialization in Lua". Archived from the original on 2014-03-06. Retrieved 2020-09-05. <a href="https://blog.cloudflare.com/introducing-lua-capnproto-better-serialization-in-lua/" target="_blank">https://blog.cloudflare.com/introducing-lua-capnproto-better-serialization-in-lua/</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></li>
</ol>

Cap'n Proto open-in-new

Cap'n Proto