Stacker: An Example Of Using LLVM

+ +

+ -

This document is another way to learn about LLVM. Unlike the LLVM Reference Manual or -LLVM Programmer's Manual, we learn +LLVM Programmer's Manual, here we learn about LLVM through the experience of creating a simple programming language named Stacker. Stacker was invented specifically as a demonstration of LLVM. The emphasis in this document is not on describing the -intricacies of LLVM itself, but on how to use it to build your own +intricacies of LLVM itself but on how to use it to build your own compiler system.

@@ -70,19 +69,18 @@ compiler system.

Amongst other things, LLVM is a platform for compiler writers. Because of its exceptionally clean and small IR (intermediate representation), compiler writing with LLVM is much easier than with -other system. As proof, the author of Stacker wrote the entire -compiler (language definition, lexer, parser, code generator, etc.) in -about four days! That's important to know because it shows -how quickly you can get a new -language up when using LLVM. Furthermore, this was the first +other system. As proof, I wrote the entire compiler (language definition, +lexer, parser, code generator, etc.) in about four days! +That's important to know because it shows how quickly you can get a new +language running when using LLVM. Furthermore, this was the first language the author ever created using LLVM. The learning curve is included in that four days.

The language described here, Stacker, is Forth-like. Programs -are simple collections of word definitions and the only thing definitions +are simple collections of word definitions, and the only thing definitions can do is manipulate a stack or generate I/O. Stacker is not a "real" -programming language; its very simple. Although it is computationally +programming language; it's very simple. Although it is computationally complete, you wouldn't use it for your next big project. However, -the fact that it is complete, its simple, and it doesn't have +the fact that it is complete, it's simple, and it doesn't have a C-like syntax make it useful for demonstration purposes. It shows that LLVM could be applied to a wide variety of languages.

The basic notions behind stacker is very simple. There's a stack of @@ -96,11 +94,11 @@ program in Stacker:

: MAIN hello_world ;

This has two "definitions" (Stacker manipulates words, not functions and words have definitions): MAIN and -hello_world. The MAIN definition is standard, it +hello_world. The MAIN definition is standard; it tells Stacker where to start. Here, MAIN is defined to simply invoke the word hello_world. The hello_world definition tells stacker to push the -"Hello, World!" string onto the stack, print it out +"Hello, World!" string on to the stack, print it out (>s), pop it off the stack (DROP), and finally print a carriage return (CR). Although hello_world uses the stack, its net effect is null. Well @@ -124,44 +122,43 @@ learned. Those lessons are described in the following subsections.

Although I knew that LLVM uses a Single Static Assignment (SSA) format, it wasn't obvious to me how prevalent this idea was in LLVM until I really started using it. Reading the -Programmer's Manual and Language Reference +Programmer's Manual and Language Reference, I noted that most of the important LLVM IR (Intermediate Representation) C++ classes were derived from the Value class. The full power of that simple design only became fully understood once I started constructing executable expressions for Stacker.

This really makes your programming go faster. Think about compiling code for the following C/C++ expression: (a|b)*((x+1)/(y+1)). Assuming the values are on the stack in the order a, b, x, y, this could be expressed in stacker as: 1 + SWAP 1 + / ROT2 OR *. -You could write a function using LLVM that computes this expression like this:


+You could write a function using LLVM that computes this expression like 
+this: 
+
+ Value* 
-expression(BasicBlock*bb, Value* a, Value* b, Value* x, Value* y )
+expression(BasicBlock* bb, Value* a, Value* b, Value* x, Value* y )
 {
-    Instruction* tail = bb->getTerminator();
-    ConstantSInt* one = ConstantSInt::get( Type::IntTy, 1);
-    BinaryOperator* or1 = 
-	BinaryOperator::create( Instruction::Or, a, b, "", tail );
-    BinaryOperator* add1 = 
-	BinaryOperator::create( Instruction::Add, x, one, "", tail );
-    BinaryOperator* add2 =
-	BinaryOperator::create( Instruction::Add, y, one, "", tail );
-    BinaryOperator* div1 = 
-	BinaryOperator::create( Instruction::Div, add1, add2, "", tail);
-    BinaryOperator* mult1 = 
-	BinaryOperator::create( Instruction::Mul, or1, div1, "", tail );
-
+    ConstantInt* one = ConstantInt::get(Type::IntTy, 1);
+    BinaryOperator* or1 = BinaryOperator::createOr(a, b, "", bb);
+    BinaryOperator* add1 = BinaryOperator::createAdd(x, one, "", bb);
+    BinaryOperator* add2 = BinaryOperator::createAdd(y, one, "", bb);
+    BinaryOperator* div1 = BinaryOperator::createDiv(add1, add2, "", bb);
+    BinaryOperator* mult1 = BinaryOperator::createMul(or1, div1, "", bb);
     return mult1;
 }
-
-"Okay, big deal," you say.  It is a big deal. Here's why. Note that I didn't
+

"Okay, big deal," you say? It is a big deal. Here's why. Note that I didn't have to tell this function which kinds of Values are being passed in. They could be -Instructions, Constants, GlobalVariables, -etc. Furthermore, if you specify Values that are incorrect for this sequence of +Instructions, Constants, GlobalVariables, or +any of the other subclasses of Value that LLVM supports. +Furthermore, if you specify Values that are incorrect for this sequence of operations, LLVM will either notice right away (at compilation time) or the LLVM -Verifier will pick up the inconsistency when the compiler runs. In no case will -you make a type error that gets passed through to the generated program. -This really helps you write a compiler that always generates correct code!

+Verifier will pick up the inconsistency when the compiler runs. In either case +LLVM prevents you from making a type error that gets passed through to the +generated program. This really helps you write a compiler that +always generates correct code!

The second point is that we don't have to worry about branching, registers, stack variables, saving partial results, etc. The instructions we create are the values we use. Note that all that was created in the above @@ -200,8 +197,8 @@ should be constructed. In general, here's what I learned:

To make this clear, consider the typical if-then-else statement (see StackerCompiler::handle_if() method). We can set this up in a single function using LLVM in the following way:

Presumably in the foregoing, the calls to the "fill_in" method would add the instructions for the "then" and "else" parts. They would use the third part of the idiom almost exclusively (inserting new instructions before the terminator). Furthermore, they could even recurse back to handle_if -should they encounter another if/then/else statement and it will just work.

Note how cleanly this all works out. In particular, the push_back methods on the BasicBlock's instruction list. These are lists of type -Instruction which also happen to be Values. To create +Instruction (which is also of type Value). To create the "if" branch we merely instantiate a BranchInst that takes as -arguments the blocks to branch to and the condition to branch on. The blocks -act like branch labels! This new BranchInst terminates -the BasicBlock provided as an argument. To give the caller a way -to keep inserting after calling handle_if we create an "exit" block -which is returned to the caller. Note that the "exit" block is used as the -terminator for both the "then" and the "else" blocks. This guarantees that no -matter what else "handle_if" or "fill_in" does, they end up at the "exit" block. +arguments the blocks to branch to and the condition to branch on. The +BasicBlock objects act like branch labels! This new +BranchInst terminates the BasicBlock provided +as an argument. To give the caller a way to keep inserting after calling +handle_if, we create an exit_bb block which is +returned +to the caller. Note that the exit_bb block is used as the +terminator for both the then_bb and the else_bb +blocks. This guarantees that no matter what else handle_if +or fill_in does, they end up at the exit_bb block.

It took a little getting used to and several rounds of postings to the LLVM -mail list to wrap my head around this instruction correctly. Even though I had +mailing list to wrap my head around this instruction correctly. Even though I had read the Language Reference and Programmer's Manual a couple times each, I still missed a few very key points:

GetElementPtrInst gives you back a Value for the last thing indexed -
All global variables in LLVM are pointers. -
Pointers must also be dereferenced with the GetElementPtrInst instruction. +
GetElementPtrInst gives you back a Value for the last thing indexed.
All global variables in LLVM are pointers.
Pointers must also be dereferenced with the GetElementPtrInst +instruction.

This means that when you look up an element in the global variable (assuming -its a struct or array), you must deference the pointer first! For many +it's a struct or array), you must deference the pointer first! For many things, this leads to the idiom:


-std::vector index_vector;
-index_vector.push_back( ConstantSInt::get( Type::LongTy, 0 );
++std::vector<Value*> index_vector;
+index_vector.push_back( ConstantInt::get( Type::LongTy, 0 );
 // ... push other indices ...
-GetElementPtrInst* gep = new GetElementPtrInst( ptr, index_vector );
-
+GetElementPtrInst* gep = GetElementPtrInst::Create( ptr, index_vector );
+

For example, suppose we have a global variable whose type is [24 x int]. The variable itself represents a pointer to that array. To subscript the array, we need two indices, not just one. The first index (0) dereferences the @@ -318,23 +319,23 @@ pointer. The second index subscripts the array. If you're a "C" programmer, this will run against your grain because you'll naturally think of the global array variable and the address of its first element as the same. That tripped me up for a while until I realized that they really do differ .. by type. -Remember that LLVM is a strongly typed language itself. Everything -has a type. The "type" of the global variable is [24 x int]*. That is, its +Remember that LLVM is strongly typed. Everything has a type. +The "type" of the global variable is [24 x int]*. That is, it's a pointer to an array of 24 ints. When you dereference that global variable with a single (0) index, you now have a "[24 x int]" type. Although the pointer value of the dereferenced global and the address of the zero'th element in the array will be the same, they differ in their type. The zero'th element has type "int" while the pointer value has type "[24 x int]".

Get this one aspect of LLVM right in your head and you'll save yourself +

Get this one aspect of LLVM right in your head, and you'll save yourself a lot of compiler writing headaches down the road.

Linkage types in LLVM can be a little confusing, especially if your compiler -writing mind has affixed very hard concepts to particular words like "weak", +writing mind has affixed firm concepts to particular words like "weak", "external", "global", "linkonce", etc. LLVM does not use the precise -definitions of say ELF or GCC even though they share common terms. To be fair, +definitions of, say, ELF or GCC, even though they share common terms. To be fair, the concepts are related and similar but not precisely the same. This can lead you to think you know what a linkage type represents but in fact it is slightly different. I recommend you read the @@ -342,16 +343,19 @@ different. I recommend you read the carefully. Then, read it again.

Here are some handy tips that I discovered along the way:

Unitialized means external. That is, the symbol is declared in the current - module and can be used by that module but it is not defined by that module.
Setting an initializer changes a global's linkage type from whatever it was - to a normal, defind global (not external). You'll need to call the setLinkage() - method to reset it if you specify the initializer after the GlobalValue has been - constructed. This is important for LinkOnce and Weak linkage types.
Appending linkage can be used to keep track of compilation information at - runtime. It could be used, for example, to build a full table of all the C++ - virtual tables or hold the C++ RTTI data, or whatever. Appending linkage can - only be applied to arrays. The arrays are concatenated together at link time.
Uninitialized means external. That is, the symbol is declared in the current + module and can be used by that module, but it is not defined by that module.
Setting an initializer changes a global' linkage type. Setting an + initializer changes a global's linkage type from whatever it was to a normal, + defined global (not external). You'll need to call the setLinkage() method to + reset it if you specify the initializer after the GlobalValue has been constructed. + This is important for LinkOnce and Weak linkage types.
Appending linkage can keep track of things. Appending linkage can + be used to keep track of compilation information at runtime. It could be used, + for example, to build a full table of all the C++ virtual tables or hold the + C++ RTTI data, or whatever. Appending linkage can only be applied to arrays. + All arrays with the same name in each module are concatenated together at link + time.

Manipulating the stack can be quite hazardous. There is no distinction given and no checking for the various types of values that can be placed on the stack. Automatic coercion between types is performed. In many -cases this is useful. For example, a boolean value placed on the stack +cases, this is useful. For example, a boolean value placed on the stack can be interpreted as an integer with good results. However, using a word that interprets that boolean value as a pointer to a string to print out will almost always yield a crash. Stacker simply leaves it @@ -406,9 +410,9 @@ is terminated by a semi-colon.

The name is up to you but it must start with a letter and contain -only letters numbers and underscore. Names are case sensitive and must not be +only letters, numbers, and underscore. Names are case sensitive and must not be the same as the name of a built-in word. The ... is replaced by -the stack manipulting words that you wish define name as.

There are three kinds of literal values in Stacker. Integer, Strings, +

There are three kinds of literal values in Stacker: Integers, Strings, and Booleans. In each case, the stack operation is to simply push the - value onto the stack. So, for example:
+ value on to the stack. So, for example:
42 " is the answer." TRUE
- will push three values onto the stack: the integer 42, the - string " is the answer." and the boolean TRUE.

+ will push three values on to the stack: the integer 42, the + string " is the answer.", and the boolean TRUE.

Words in a definition come in two flavors: built-in and programmer defined. Simply mentioning the name of a previously defined or declared -programmer-defined word causes that word's definition to be invoked. It +programmer-defined word causes that word's stack actions to be invoked. It is somewhat like a function call in other languages. The built-in -words have various effects, described below.

Sometimes you need to call a word before it is defined. For this, you can use the FORWARD declaration. It looks like this:

The built-in words of the Stacker language are put in several groups depending on what they do. The groups are as follows:

LogicalThese words provide the logical operations for +
Logical: These words provide the logical operations for comparing stack operands.
The words are: < > <= >= = <> true false.
BitwiseThese words perform bitwise computations on +
Bitwise: These words perform bitwise computations on their operands.
The words are: << >> XOR AND NOT
ArithmeticThese words perform arithmetic computations on +
Arithmetic: These words perform arithmetic computations on their operands.
The words are: ABS NEG + - * / MOD */ ++ -- MIN MAX
StackThese words manipulate the stack directly by moving - its elements around.
The words are: DROP DUP SWAP OVER ROT DUP2 DROP2 PICK TUCK
MemoryThese words allocate, free and manipulate memory + its elements around.
The words are: DROP DROP2 NIP NIP2 DUP DUP2 + SWAP SWAP2 OVER OVER2 ROT ROT2 RROT RROT2 TUCK TUCK2 PICK SELECT ROLL
MemoryThese words allocate, free, and manipulate memory areas outside the stack.
The words are: MALLOC FREE GET PUT
ControlThese words alter the normal left to right flow +
Control: These words alter the normal left to right flow of execution.
The words are: IF ELSE ENDIF WHILE END RETURN EXIT RECURSE
I/O These words perform output on the standard output +
I/O: These words perform output on the standard output and input on the standard input. No other I/O is possible in Stacker.
The words are: SPACE TAB CR >s >d >c <s <d <c.

@@ -500,12 +510,18 @@ using the following construction:

p - a pointer to a malloc'd memory block

- - - - - +

Definition Of Operation Of Built In Words
LOGICAL OPERATIONS
Word	Name	Operation	Description
<

+ + + + + + + + + + - + - + + + + + + + + - - @@ -598,84 +619,94 @@ using the following construction:

are bitwise exclusive OR'd together and pushed back on the stack. For example, The sequence 1 3 XOR yields 2. - - + + + + + + + + on to the stack + on to the stack + on to the stack + on to the stack + of w1 by w2 is pushed back on to the stack + divided by w3. The result is pushed back on to the stack. + pushed back on to the stack. + pushed back on to the stack. + on to the stack. + on to the stack. + + + + + + + - - @@ -703,7 +734,7 @@ using the following construction:

- @@ -717,7 +748,7 @@ using the following construction:

@@ -725,27 +756,27 @@ using the following construction:

- @@ -756,7 +787,7 @@ using the following construction:

- + @@ -814,15 +845,20 @@ using the following construction:

how much to rotate. That is, ROLL with n=1 is the same as ROT and ROLL with n=2 is the same as ROT2. - - + + + + + + + + block is pushed on to the stack. @@ -862,8 +898,13 @@ using the following construction:

pushed back on the stack so this doesn't count as a "use ptr" in the FREE idiom. - - + + + + + + + @@ -905,27 +946,36 @@ using the following construction:

executed. In either case, after the (words....) have executed, execution continues immediately following the ENDIF. - - + + - - - - + 10 WHILE >d -- END
+ This will print the numbers from 10 down to 1. 10 is pushed on the + stack. Since that is non-zero, the while loop is entered. The top of + the stack (10) is printed out with >d. The top of the stack is + decremented, yielding 9 and control is transfered back to the WHILE + keyword. The process starts all over again and repeats until + the top of stack is decremented to 0 at which point the WHILE test + fails and control is transfered to the word after the END. + + + + + + + + + @@ -949,30 +999,32 @@ using the following construction:

- + - + - + - + - + @@ -982,16 +1034,17 @@ using the following construction:

to see instantly the net effect of the definition.

Definition Of Operation Of Built In Words
LOGICAL OPERATIONS
Word	Name	Operation	Description
<	LT	w1 w2 -- b	Two values (w1 and w2) are popped off the stack and @@ -554,15 +570,20 @@ using the following construction:
FALSE	FALSE	-- b	The boolean value FALSE (0) is pushed onto the stack.	The boolean value FALSE (0) is pushed on to the stack.
TRUE	TRUE	-- b	The boolean value TRUE (-1) is pushed onto the stack.	The boolean value TRUE (-1) is pushed on to the stack.
BITWISE OPERATORS
Word	Name	Operation	Description
BITWISE OPERATIONS
Word	Name	Operation	Description
<<	SHL	w1 w2 -- w1<<w2
ARITHMETIC OPERATIONS
Word	Name	Operation	Description
ARITHMETIC OPERATORS
Word	Name	Operation	Description
ABS	ABS	w -- \|w\|	One value s popped off the stack; its absolute value is computed - and then pushed onto the stack. If w1 is -1 then w2 is 1. If w1 is + and then pushed on to the stack. If w1 is -1 then w2 is 1. If w1 is 1 then w2 is also 1.
NEG	NEG	w -- -w	One value is popped off the stack which is negated and then - pushed back onto the stack. If w1 is -1 then w2 is 1. If w1 is + pushed back on to the stack. If w1 is -1 then w2 is 1. If w1 is 1 then w2 is -1.
+	ADD	w1 w2 -- w2+w1	Two values are popped off the stack. Their sum is pushed back - onto the stack
-	SUB	w1 w2 -- w2-w1	Two values are popped off the stack. Their difference is pushed back - onto the stack
*	MUL	w1 w2 -- w2*w1	Two values are popped off the stack. Their product is pushed back - onto the stack
/	DIV	w1 w2 -- w2/w1	Two values are popped off the stack. Their quotient is pushed back - onto the stack
MOD	MOD	w1 w2 -- w2%w1	Two values are popped off the stack. Their remainder after division - of w1 by w2 is pushed back onto the stack
*/	STAR_SLAH	w1 w2 w3 -- (w3*w2)/w1	Three values are popped off the stack. The product of w1 and w2 is - divided by w3. The result is pushed back onto the stack.
++	INCR	w -- w+1	One value is popped off the stack. It is incremented by one and then - pushed back onto the stack.
--	DECR	w -- w-1	One value is popped off the stack. It is decremented by one and then - pushed back onto the stack.
MIN	MIN	w1 w2 -- (w2<w1?w2:w1)	Two values are popped off the stack. The larger one is pushed back - onto the stack.
MAX	MAX	w1 w2 -- (w2>w1?w2:w1)	Two values are popped off the stack. The larger value is pushed back - onto the stack.
STACK MANIPULATION OPERATORS
Word	Name	Operation	Description
STACK MANIPULATION OPERATIONS
Word	Name	Operation	Description
DROP	DROP	w --
DUP	DUP	w1 -- w1 w1	One value is popped off the stack. That value is then pushed onto +	One value is popped off the stack. That value is then pushed on to the stack twice to duplicate the top stack vaue.
DUP2	SWAP	w1 w2 -- w2 w1	The top two stack items are reversed in their order. That is, two - values are popped off the stack and pushed back onto the stack in + values are popped off the stack and pushed back on to the stack in the opposite order they were popped.
SWAP2	w1 w2 w3 w4 -- w3 w4 w2 w1	The top four stack items are swapped in pairs. That is, two values are popped and retained. Then, two more values are popped and retained. - The values are pushed back onto the stack in the reverse order but - in pairs. + The values are pushed back on to the stack in the reverse order but + in pairs.
OVER	OVER	w1 w2-- w1 w2 w1	Two values are popped from the stack. They are pushed back - onto the stack in the order w1 w2 w1. This seems to cause the + on to the stack in the order w1 w2 w1. This seems to cause the top stack element to be duplicated "over" the next value.
OVER2	OVER2	w1 w2 w3 w4 -- w1 w2 w3 w4 w1 w2	The third and fourth values on the stack are replicated onto the +	The third and fourth values on the stack are replicated on to the top of the stack
ROT	ROT	w1 w2 w3 -- w2 w3 w1	The top three values are rotated. That is, three value are popped - off the stack. They are pushed back onto the stack in the order + off the stack. They are pushed back on to the stack in the order w1 w3 w2.
ROT2
RROT	RROT	w1 w2 w3 -- w2 w3 w1	w1 w2 w3 -- w3 w1 w2	Reverse rotation. Like ROT, but it rotates the other way around. Essentially, the third element on the stack is moved to the top of the stack.
MEMORY OPERATIONS
Word	Name	Operation	Description
MEMORY OPERATORS
Word	Name	Operation	Description
MALLOC	MALLOC	w1 -- p	One value is popped off the stack. The value is used as the size of a memory block to allocate. The size is in bytes, not words. The memory allocation is completed and the address of the memory - block is pushed onto the stack.
FREE	FREE
CONTROL FLOW OPERATIONS
Word	Name	Operation	Description
CONTROL FLOW OPERATORS
Word	Name	Operation	Description
RETURN	RETURN	--
WHILE (words...) END	WHILE (words...) END
WHILE word END	WHILE word END	b -- b	The boolean value on the top of the stack is examined. If it is non-zero then the - "words..." between WHILE and END are executed. Execution then begins again at the WHILE where another - boolean is popped off the stack. To prevent this operation from eating up the entire - stack, you should push onto the stack (just before the END) a boolean value that indicates - whether to terminate. Note that since booleans and integers can be coerced you can - use the following "for loop" idiom: - `(push count) WHILE (words...) -- END` +	The boolean value on the top of the stack is examined (not popped). If + it is non-zero then the "word" between WHILE and END is executed. + Execution then begins again at the WHILE where the boolean on the top of + the stack is examined again. The stack is not modified by the WHILE...END + loop, only examined. It is imperative that the "word" in the body of the + loop ensure that the top of the stack contains the next boolean to examine + when it completes. Note that since booleans and integers can be coerced + you can use the following "for loop" idiom: + `(push count) WHILE word -- END` For example: - `10 WHILE DUP >d -- END` - This will print the numbers from 10 down to 1. 10 is pushed on the stack. Since that is - non-zero, the while loop is entered. The top of the stack (10) is duplicated and then - printed out with >d. The top of the stack is decremented, yielding 9 and control is - transfered back to the WHILE keyword. The process starts all over again and repeats until - the top of stack is decremented to 0 at which the WHILE test fails and control is - transfered to the word after the END.
INPUT & OUTPUT OPERATIONS
Word	Name	Operation	Description
INPUT & OUTPUT OPERATORS
Word	Name	Operation	Description
SPACE	SPACE	--
>d	OUT_STR	--	A value is popped from the stack. It is put out as a decimal integer.	A value is popped from the stack. It is put out as a decimal + integer.
>c	OUT_CHR	--	A value is popped from the stack. It is put out as an ASCII character.	A value is popped from the stack. It is put out as an ASCII + character.
<s	IN_STR	-- s	A string is read from the input via the scanf(3) format string " %as". The - resulting string is pushed onto the stack.	A string is read from the input via the scanf(3) format string " %as". + The resulting string is pushed on to the stack.
<d	IN_STR	-- w	An integer is read from the input via the scanf(3) format string " %d". The - resulting value is pushed onto the stack	An integer is read from the input via the scanf(3) format string " %d". + The resulting value is pushed on to the stack
<c	IN_CHR	-- w	A single character is read from the input via the scanf(3) format string - " %c". The value is converted to an integer and pushed onto the stack.	A single character is read from the input via the scanf(3) format string + " %c". The value is converted to an integer and pushed on to the stack.
DUMP	DUMP