Difference between revisions of "Hypergraph Format"

From ACL Wiki
Jump to: navigation, search
(Protocol Buffers)
Line 10: Line 10:
 
Con:  
 
Con:  
 
* Space inefficiency
 
* Space inefficiency
 +
* Requires custom parser for speed
  
 
== Protocol Buffers ==
 
== Protocol Buffers ==
Line 21: Line 22:
 
* Very fast to read (particularly in C++ and Java, hopefully soon in python)
 
* Very fast to read (particularly in C++ and Java, hopefully soon in python)
 
* Very space efficient
 
* Very space efficient
* Implementations in every language (although requires a separate library)
+
* Implementations in Java, C++ and Python
 
* Automatically generates typed stubs  
 
* Automatically generates typed stubs  
  
 
Con:
 
Con:
 +
* No implementations for Perl, C#, or other languages commonly used by NLP folks
 +
* Requires a separate library; adds an external dependency to spec
 
* "It's really easy to get up to some of the data size limits that are in place to prevent malicious data from having the PB parser allocate too much memory". Some of the limits are described in the section describing SetTotalBytesLimit on [http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.io.coded_stream.html this page].  
 
* "It's really easy to get up to some of the data size limits that are in place to prevent malicious data from having the PB parser allocate too much memory". Some of the limits are described in the section describing SetTotalBytesLimit on [http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.io.coded_stream.html this page].  
 
* "You typically have to create a full hypergraph protocol buffer object before you can serialize it, so you either have to use the PB data structures internally in your code or you have to copy your data structure. While doing this copy, you can end up with two copies of the forest in memory, which is bad for memory usage."
 
* "You typically have to create a full hypergraph protocol buffer object before you can serialize it, so you either have to use the PB data structures internally in your code or you have to copy your data structure. While doing this copy, you can end up with two copies of the forest in memory, which is bad for memory usage."

Revision as of 10:18, 8 November 2010

JSON

JSON Description

Pro:

  • Implementations in every language (often packaged with language).
  • Human readable
  • Already used in CDec for forest output

Con:

  • Space inefficiency
  • Requires custom parser for speed

Protocol Buffers

Protocol Buffer Description

Implementation Sketch

Pro:

  • Conversion to and from JSON (protobuf-json)
  • Very fast to read (particularly in C++ and Java, hopefully soon in python)
  • Very space efficient
  • Implementations in Java, C++ and Python
  • Automatically generates typed stubs

Con:

  • No implementations for Perl, C#, or other languages commonly used by NLP folks
  • Requires a separate library; adds an external dependency to spec
  • "It's really easy to get up to some of the data size limits that are in place to prevent malicious data from having the PB parser allocate too much memory". Some of the limits are described in the section describing SetTotalBytesLimit on this page.
  • "You typically have to create a full hypergraph protocol buffer object before you can serialize it, so you either have to use the PB data structures internally in your code or you have to copy your data structure. While doing this copy, you can end up with two copies of the forest in memory, which is bad for memory usage."

Variation of SLF (Standard Lattice Format)

SLF Specification

Pro:

  • Blindingly fast.
  • Could be implemented to work lazy/streaming.

Con:

  • Requires a custom format
  • Probably need specialized language bindings.

Tiburon Format

Tiburon Specification