11 YAML is a human readable data serialization language. The full YAML language
12 spec can be read at `yaml.org
13 <http://www.yaml.org/spec/1.2/spec.html#Introduction>`_. The simplest form of
14 yaml is just "scalars", "mappings", and "sequences". A scalar is any number
15 or string. The pound/hash symbol (#) begins a comment line. A mapping is
16 a set of key-value pairs where the key ends with a colon. For example:
24 A sequence is a list of items where each item starts with a leading dash ('-').
34 You can combine mappings and sequences by indenting. For example a sequence
35 of mappings in which one of the mapping values is itself a sequence:
39 # a sequence of mappings with one key's value being a sequence
52 Sometime sequences are known to be short and the one entry per line is too
53 verbose, so YAML offers an alternate syntax for sequences called a "Flow
54 Sequence" in which you put comma separated sequence elements into square
55 brackets. The above example could then be simplified to :
60 # a sequence of mappings with one key's value being a flow sequence
66 cpus: [ PowerPC, x86 ]
69 Introduction to YAML I/O
70 ========================
72 The use of indenting makes the YAML easy for a human to read and understand,
73 but having a program read and write YAML involves a lot of tedious details.
74 The YAML I/O library structures and simplifies reading and writing YAML
77 YAML I/O assumes you have some "native" data structures which you want to be
78 able to dump as YAML and recreate from YAML. The first step is to try
79 writing example YAML for your data structures. You may find after looking at
80 possible YAML representations that a direct mapping of your data structures
81 to YAML is not very readable. Often the fields are not in the order that
82 a human would find readable. Or the same information is replicated in multiple
83 locations, making it hard for a human to write such YAML correctly.
85 In relational database theory there is a design step called normalization in
86 which you reorganize fields and tables. The same considerations need to
87 go into the design of your YAML encoding. But, you may not want to change
88 your existing native data structures. Therefore, when writing out YAML
89 there may be a normalization step, and when reading YAML there would be a
90 corresponding denormalization step.
92 YAML I/O uses a non-invasive, traits based design. YAML I/O defines some
93 abstract base templates. You specialize those templates on your data types.
94 For instance, if you have an enumerated type FooBar you could specialize
95 ScalarEnumerationTraits on that type and define the enumeration() method:
99 using llvm::yaml::ScalarEnumerationTraits;
100 using llvm::yaml::IO;
103 struct ScalarEnumerationTraits<FooBar> {
104 static void enumeration(IO &io, FooBar &value) {
110 As with all YAML I/O template specializations, the ScalarEnumerationTraits is used for
111 both reading and writing YAML. That is, the mapping between in-memory enum
112 values and the YAML string representation is only in one place.
113 This assures that the code for writing and parsing of YAML stays in sync.
115 To specify a YAML mappings, you define a specialization on
116 llvm::yaml::MappingTraits.
117 If your native data structure happens to be a struct that is already normalized,
118 then the specialization is simple. For example:
122 using llvm::yaml::MappingTraits;
123 using llvm::yaml::IO;
126 struct MappingTraits<Person> {
127 static void mapping(IO &io, Person &info) {
128 io.mapRequired("name", info.name);
129 io.mapOptional("hat-size", info.hatSize);
134 A YAML sequence is automatically inferred if you data type has begin()/end()
135 iterators and a push_back() method. Therefore any of the STL containers
136 (such as std::vector<>) will automatically translate to YAML sequences.
138 Once you have defined specializations for your data types, you can
139 programmatically use YAML I/O to write a YAML document:
143 using llvm::yaml::Output;
151 std::vector<Person> persons;
152 persons.push_back(tom);
153 persons.push_back(dan);
155 Output yout(llvm::outs());
158 This would write the following:
167 And you can also read such YAML documents with the following code:
171 using llvm::yaml::Input;
173 typedef std::vector<Person> PersonList;
174 std::vector<PersonList> docs;
176 Input yin(document.getBuffer());
182 // Process read document
183 for ( PersonList &pl : docs ) {
184 for ( Person &person : pl ) {
185 cout << "name=" << person.name;
189 One other feature of YAML is the ability to define multiple documents in a
190 single file. That is why reading YAML produces a vector of your document type.
197 When parsing a YAML document, if the input does not match your schema (as
198 expressed in your XxxTraits<> specializations). YAML I/O
199 will print out an error message and your Input object's error() method will
200 return true. For instance the following document:
209 Has a key (shoe-size) that is not defined in the schema. YAML I/O will
210 automatically generate this error:
214 YAML:2:2: error: unknown key 'shoe-size'
218 Similar errors are produced for other input not conforming to the schema.
224 YAML scalars are just strings (i.e. not a sequence or mapping). The YAML I/O
225 library provides support for translating between YAML scalars and specific
231 The following types have built-in support in YAML I/O:
247 That is, you can use those types in fields of MappingTraits or as element type
248 in sequence. When reading, YAML I/O will validate that the string found
249 is convertible to that type and error out if not.
254 Given that YAML I/O is trait based, the selection of how to convert your data
255 to YAML is based on the type of your data. But in C++ type matching, typedefs
256 do not generate unique type names. That means if you have two typedefs of
257 unsigned int, to YAML I/O both types look exactly like unsigned int. To
258 facilitate make unique type names, YAML I/O provides a macro which is used
259 like a typedef on built-in types, but expands to create a class with conversion
260 operators to and from the base type. For example:
264 LLVM_YAML_STRONG_TYPEDEF(uint32_t, MyFooFlags)
265 LLVM_YAML_STRONG_TYPEDEF(uint32_t, MyBarFlags)
267 This generates two classes MyFooFlags and MyBarFlags which you can use in your
268 native data structures instead of uint32_t. They are implicitly
269 converted to and from uint32_t. The point of creating these unique types
270 is that you can now specify traits on them to get different YAML conversions.
274 An example use of a unique type is that YAML I/O provides fixed sized unsigned
275 integers that are written with YAML I/O as hexadecimal instead of the decimal
276 format used by the built-in integer types:
283 You can use llvm::yaml::Hex32 instead of uint32_t and the only different will
284 be that when YAML I/O writes out that type it will be formatted in hexadecimal.
287 ScalarEnumerationTraits
288 -----------------------
289 YAML I/O supports translating between in-memory enumerations and a set of string
290 values in YAML documents. This is done by specializing ScalarEnumerationTraits<>
291 on your enumeration type and define a enumeration() method.
292 For instance, suppose you had an enumeration of CPUs and a struct with it as
308 To support reading and writing of this enumeration, you can define a
309 ScalarEnumerationTraits specialization on CPUs, which can then be used
314 using llvm::yaml::ScalarEnumerationTraits;
315 using llvm::yaml::MappingTraits;
316 using llvm::yaml::IO;
319 struct ScalarEnumerationTraits<CPUs> {
320 static void enumeration(IO &io, CPUs &value) {
321 io.enumCase(value, "x86_64", cpu_x86_64);
322 io.enumCase(value, "x86", cpu_x86);
323 io.enumCase(value, "PowerPC", cpu_PowerPC);
328 struct MappingTraits<Info> {
329 static void mapping(IO &io, Info &info) {
330 io.mapRequired("cpu", info.cpu);
331 io.mapOptional("flags", info.flags, 0);
335 When reading YAML, if the string found does not match any of the the strings
336 specified by enumCase() methods, an error is automatically generated.
337 When writing YAML, if the value being written does not match any of the values
338 specified by the enumCase() methods, a runtime assertion is triggered.
343 Another common data structure in C++ is a field where each bit has a unique
344 meaning. This is often used in a "flags" field. YAML I/O has support for
345 converting such fields to a flow sequence. For instance suppose you
346 had the following bit flags defined:
357 LLVM_YAML_STRONG_TYPEDEF(uint32_t, MyFlags)
359 To support reading and writing of MyFlags, you specialize ScalarBitSetTraits<>
360 on MyFlags and provide the bit values and their names.
364 using llvm::yaml::ScalarBitSetTraits;
365 using llvm::yaml::MappingTraits;
366 using llvm::yaml::IO;
369 struct ScalarBitSetTraits<MyFlags> {
370 static void bitset(IO &io, MyFlags &value) {
371 io.bitSetCase(value, "hollow", flagHollow);
372 io.bitSetCase(value, "flat", flagFlat);
373 io.bitSetCase(value, "round", flagRound);
374 io.bitSetCase(value, "pointy", flagPointy);
384 struct MappingTraits<Info> {
385 static void mapping(IO &io, Info& info) {
386 io.mapRequired("name", info.name);
387 io.mapRequired("flags", info.flags);
391 With the above, YAML I/O (when writing) will test mask each value in the
392 bitset trait against the flags field, and each that matches will
393 cause the corresponding string to be added to the flow sequence. The opposite
394 is done when reading and any unknown string values will result in a error. With
395 the above schema, a same valid YAML document is:
400 flags: [ pointy, flat ]
405 Sometimes for readability a scalar needs to be formatted in a custom way. For
406 instance your internal data structure may use a integer for time (seconds since
407 some epoch), but in YAML it would be much nicer to express that integer in
408 some time format (e.g. 4-May-2012 10:30pm). YAML I/O has a way to support
409 custom formatting and parsing of scalar types by specializing ScalarTraits<> on
410 your data type. When writing, YAML I/O will provide the native type and
411 your specialization must create a temporary llvm::StringRef. When reading,
412 YAML I/O will provide an llvm::StringRef of scalar and your specialization
413 must convert that to your native data type. An outline of a custom scalar type
418 using llvm::yaml::ScalarTraits;
419 using llvm::yaml::IO;
422 struct ScalarTraits<MyCustomType> {
423 static void output(const T &value, llvm::raw_ostream &out) {
424 out << value; // do custom formatting here
426 static StringRef input(StringRef scalar, T &value) {
427 // do custom parsing here. Return the empty string on success,
428 // or an error message on failure.
431 // Determine if this scalar needs quotes.
432 static bool mustQuote(StringRef) { return true; }
439 To be translated to or from a YAML mapping for your type T you must specialize
440 llvm::yaml::MappingTraits on T and implement the "void mapping(IO &io, T&)"
441 method. If your native data structures use pointers to a class everywhere,
442 you can specialize on the class pointer. Examples:
446 using llvm::yaml::MappingTraits;
447 using llvm::yaml::IO;
449 // Example of struct Foo which is used by value
451 struct MappingTraits<Foo> {
452 static void mapping(IO &io, Foo &foo) {
453 io.mapOptional("size", foo.size);
458 // Example of struct Bar which is natively always a pointer
460 struct MappingTraits<Bar*> {
461 static void mapping(IO &io, Bar *&bar) {
462 io.mapOptional("size", bar->size);
471 The mapping() method is responsible, if needed, for normalizing and
472 denormalizing. In a simple case where the native data structure requires no
473 normalization, the mapping method just uses mapOptional() or mapRequired() to
474 bind the struct's fields to YAML key names. For example:
478 using llvm::yaml::MappingTraits;
479 using llvm::yaml::IO;
482 struct MappingTraits<Person> {
483 static void mapping(IO &io, Person &info) {
484 io.mapRequired("name", info.name);
485 io.mapOptional("hat-size", info.hatSize);
493 When [de]normalization is required, the mapping() method needs a way to access
494 normalized values as fields. To help with this, there is
495 a template MappingNormalization<> which you can then use to automatically
496 do the normalization and denormalization. The template is used to create
497 a local variable in your mapping() method which contains the normalized keys.
499 Suppose you have native data type
500 Polar which specifies a position in polar coordinates (distance, angle):
509 but you've decided the normalized YAML for should be in x,y coordinates. That
510 is, you want the yaml to look like:
517 You can support this by defining a MappingTraits that normalizes the polar
518 coordinates to x,y coordinates when writing YAML and denormalizes x,y
519 coordinates into polar when reading YAML.
523 using llvm::yaml::MappingTraits;
524 using llvm::yaml::IO;
527 struct MappingTraits<Polar> {
529 class NormalizedPolar {
531 NormalizedPolar(IO &io)
534 NormalizedPolar(IO &, Polar &polar)
535 : x(polar.distance * cos(polar.angle)),
536 y(polar.distance * sin(polar.angle)) {
538 Polar denormalize(IO &) {
539 return Polar(sqrt(x*x+y*y), arctan(x,y));
546 static void mapping(IO &io, Polar &polar) {
547 MappingNormalization<NormalizedPolar, Polar> keys(io, polar);
549 io.mapRequired("x", keys->x);
550 io.mapRequired("y", keys->y);
554 When writing YAML, the local variable "keys" will be a stack allocated
555 instance of NormalizedPolar, constructed from the supplied polar object which
556 initializes it x and y fields. The mapRequired() methods then write out the x
557 and y values as key/value pairs.
559 When reading YAML, the local variable "keys" will be a stack allocated instance
560 of NormalizedPolar, constructed by the empty constructor. The mapRequired
561 methods will find the matching key in the YAML document and fill in the x and y
562 fields of the NormalizedPolar object keys. At the end of the mapping() method
563 when the local keys variable goes out of scope, the denormalize() method will
564 automatically be called to convert the read values back to polar coordinates,
565 and then assigned back to the second parameter to mapping().
567 In some cases, the normalized class may be a subclass of the native type and
568 could be returned by the denormalize() method, except that the temporary
569 normalized instance is stack allocated. In these cases, the utility template
570 MappingNormalizationHeap<> can be used instead. It just like
571 MappingNormalization<> except that it heap allocates the normalized object
572 when reading YAML. It never destroys the normalized object. The denormalize()
573 method can this return "this".
578 Within a mapping() method, calls to io.mapRequired() mean that that key is
579 required to exist when parsing YAML documents, otherwise YAML I/O will issue an
582 On the other hand, keys registered with io.mapOptional() are allowed to not
583 exist in the YAML document being read. So what value is put in the field
584 for those optional keys?
585 There are two steps to how those optional fields are filled in. First, the
586 second parameter to the mapping() method is a reference to a native class. That
587 native class must have a default constructor. Whatever value the default
588 constructor initially sets for an optional field will be that field's value.
589 Second, the mapOptional() method has an optional third parameter. If provided
590 it is the value that mapOptional() should set that field to if the YAML document
591 does not have that key.
593 There is one important difference between those two ways (default constructor
594 and third parameter to mapOptional). When YAML I/O generates a YAML document,
595 if the mapOptional() third parameter is used, if the actual value being written
596 is the same as (using ==) the default value, then that key/value is not written.
602 When writing out a YAML document, the keys are written in the order that the
603 calls to mapRequired()/mapOptional() are made in the mapping() method. This
604 gives you a chance to write the fields in an order that a human reader of
605 the YAML document would find natural. This may be different that the order
606 of the fields in the native class.
608 When reading in a YAML document, the keys in the document can be in any order,
609 but they are processed in the order that the calls to mapRequired()/mapOptional()
610 are made in the mapping() method. That enables some interesting
611 functionality. For instance, if the first field bound is the cpu and the second
612 field bound is flags, and the flags are cpu specific, you can programmatically
613 switch how the flags are converted to and from YAML based on the cpu.
614 This works for both reading and writing. For example:
618 using llvm::yaml::MappingTraits;
619 using llvm::yaml::IO;
627 struct MappingTraits<Info> {
628 static void mapping(IO &io, Info &info) {
629 io.mapRequired("cpu", info.cpu);
630 // flags must come after cpu for this to work when reading yaml
631 if ( info.cpu == cpu_x86_64 )
632 io.mapRequired("flags", *(My86_64Flags*)info.flags);
634 io.mapRequired("flags", *(My86Flags*)info.flags);
642 The YAML syntax supports tags as a way to specify the type of a node before
643 it is parsed. This allows dynamic types of nodes. But the YAML I/O model uses
644 static typing, so there are limits to how you can use tags with the YAML I/O
645 model. Recently, we added support to YAML I/O for checking/setting the optional
646 tag on a map. Using this functionality it is even possbile to support different
647 mappings, as long as they are convertable.
649 To check a tag, inside your mapping() method you can use io.mapTag() to specify
650 what the tag should be. This will also add that tag when writing yaml.
655 Sometimes in a yaml map, each key/value pair is valid, but the combination is
656 not. This is similar to something having no syntax errors, but still having
657 semantic errors. To support semantic level checking, YAML I/O allows
658 an optional ``validate()`` method in a MappingTraits template specialization.
660 When parsing yaml, the ``validate()`` method is call *after* all key/values in
661 the map have been processed. Any error message returned by the ``validate()``
662 method during input will be printed just a like a syntax error would be printed.
663 When writing yaml, the ``validate()`` method is called *before* the yaml
664 key/values are written. Any error during output will trigger an ``assert()``
665 because it is a programming error to have invalid struct values.
670 using llvm::yaml::MappingTraits;
671 using llvm::yaml::IO;
678 struct MappingTraits<Stuff> {
679 static void mapping(IO &io, Stuff &stuff) {
682 static StringRef validate(IO &io, Stuff &stuff) {
683 // Look at all fields in 'stuff' and if there
684 // are any bad values return a string describing
685 // the error. Otherwise return an empty string.
694 To be translated to or from a YAML sequence for your type T you must specialize
695 llvm::yaml::SequenceTraits on T and implement two methods:
696 ``size_t size(IO &io, T&)`` and
697 ``T::value_type& element(IO &io, T&, size_t indx)``. For example:
702 struct SequenceTraits<MySeq> {
703 static size_t size(IO &io, MySeq &list) { ... }
704 static MySeqEl &element(IO &io, MySeq &list, size_t index) { ... }
707 The size() method returns how many elements are currently in your sequence.
708 The element() method returns a reference to the i'th element in the sequence.
709 When parsing YAML, the element() method may be called with an index one bigger
710 than the current size. Your element() method should allocate space for one
711 more element (using default constructor if element is a C++ object) and returns
712 a reference to that new allocated space.
717 A YAML "flow sequence" is a sequence that when written to YAML it uses the
718 inline notation (e.g [ foo, bar ] ). To specify that a sequence type should
719 be written in YAML as a flow sequence, your SequenceTraits specialization should
720 add "static const bool flow = true;". For instance:
725 struct SequenceTraits<MyList> {
726 static size_t size(IO &io, MyList &list) { ... }
727 static MyListEl &element(IO &io, MyList &list, size_t index) { ... }
729 // The existence of this member causes YAML I/O to use a flow sequence
730 static const bool flow = true;
733 With the above, if you used MyList as the data type in your native data
734 structures, then then when converted to YAML, a flow sequence of integers
735 will be used (e.g. [ 10, -3, 4 ]).
740 Since a common source of sequences is std::vector<>, YAML I/O provides macros:
741 LLVM_YAML_IS_SEQUENCE_VECTOR() and LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR() which
742 can be used to easily specify SequenceTraits<> on a std::vector type. YAML
743 I/O does not partial specialize SequenceTraits on std::vector<> because that
744 would force all vectors to be sequences. An example use of the macros:
748 std::vector<MyType1>;
749 std::vector<MyType2>;
750 LLVM_YAML_IS_SEQUENCE_VECTOR(MyType1)
751 LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR(MyType2)
758 YAML allows you to define multiple "documents" in a single YAML file. Each
759 new document starts with a left aligned "---" token. The end of all documents
760 is denoted with a left aligned "..." token. Many users of YAML will never
761 have need for multiple documents. The top level node in their YAML schema
762 will be a mapping or sequence. For those cases, the following is not needed.
763 But for cases where you do want multiple documents, you can specify a
764 trait for you document list type. The trait has the same methods as
765 SequenceTraits but is named DocumentListTraits. For example:
770 struct DocumentListTraits<MyDocList> {
771 static size_t size(IO &io, MyDocList &list) { ... }
772 static MyDocType element(IO &io, MyDocList &list, size_t index) { ... }
778 When an llvm::yaml::Input or llvm::yaml::Output object is created their
779 constructors take an optional "context" parameter. This is a pointer to
780 whatever state information you might need.
782 For instance, in a previous example we showed how the conversion type for a
783 flags field could be determined at runtime based on the value of another field
784 in the mapping. But what if an inner mapping needs to know some field value
785 of an outer mapping? That is where the "context" parameter comes in. You
786 can set values in the context in the outer map's mapping() method and
787 retrieve those values in the inner map's mapping() method.
789 The context value is just a void*. All your traits which use the context
790 and operate on your native data types, need to agree what the context value
791 actually is. It could be a pointer to an object or struct which your various
792 traits use to shared context sensitive information.
798 The llvm::yaml::Output class is used to generate a YAML document from your
799 in-memory data structures, using traits defined on your data types.
800 To instantiate an Output object you need an llvm::raw_ostream, and optionally
805 class Output : public IO {
807 Output(llvm::raw_ostream &, void *context=NULL);
809 Once you have an Output object, you can use the C++ stream operator on it
810 to write your native data as YAML. One thing to recall is that a YAML file
811 can contain multiple "documents". If the top level data structure you are
812 streaming as YAML is a mapping, scalar, or sequence, then Output assumes you
813 are generating one document and wraps the mapping output
814 with "``---``" and trailing "``...``".
818 using llvm::yaml::Output;
820 void dumpMyMapDoc(const MyMapType &info) {
821 Output yout(llvm::outs());
825 The above could produce output like:
834 On the other hand, if the top level data structure you are streaming as YAML
835 has a DocumentListTraits specialization, then Output walks through each element
836 of your DocumentList and generates a "---" before the start of each element
837 and ends with a "...".
841 using llvm::yaml::Output;
843 void dumpMyMapDoc(const MyDocListType &docList) {
844 Output yout(llvm::outs());
848 The above could produce output like:
863 The llvm::yaml::Input class is used to parse YAML document(s) into your native
864 data structures. To instantiate an Input
865 object you need a StringRef to the entire YAML file, and optionally a context
870 class Input : public IO {
872 Input(StringRef inputContent, void *context=NULL);
874 Once you have an Input object, you can use the C++ stream operator to read
875 the document(s). If you expect there might be multiple YAML documents in
876 one file, you'll need to specialize DocumentListTraits on a list of your
877 document type and stream in that document list type. Otherwise you can
878 just stream in the document type. Also, you can check if there was
879 any syntax errors in the YAML be calling the error() method on the Input
884 // Reading a single document
885 using llvm::yaml::Input;
887 Input yin(mb.getBuffer());
889 // Parse the YAML file
900 // Reading multiple documents in one file
901 using llvm::yaml::Input;
903 LLVM_YAML_IS_DOCUMENT_LIST_VECTOR(std::vector<MyDocType>)
905 Input yin(mb.getBuffer());
907 // Parse the YAML file
908 std::vector<MyDocType> theDocList;