From: Chris Lattner Date: Mon, 22 Oct 2007 04:32:37 +0000 (+0000) Subject: add part 1, review appreciated. X-Git-Url: http://demsky.eecs.uci.edu/git/?a=commitdiff_plain;h=c38ef54f0b11dc980fc289c583ebad5264b46fff;p=oota-llvm.git add part 1, review appreciated. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@43215 91177308-0d34-0410-b5e6-96231b3b80d8 --- diff --git a/docs/tutorial/LangImpl1.html b/docs/tutorial/LangImpl1.html new file mode 100644 index 00000000000..49ad1818d8b --- /dev/null +++ b/docs/tutorial/LangImpl1.html @@ -0,0 +1,263 @@ + + + + + Kaleidoscope: The basic language, with its lexer + + + + + + + +
Kaleidoscope: The basic language, with its lexer
+ +
+

Written by Chris Lattner

+
+ + +
Tutorial Introduction
+ + +
+ +

Welcome to the "Implementing a language with LLVM" tutorial. This tutorial +will run through implementation of a simple language, showing how fun and easy +it can be. This tutorial will get you up and started and build a framework you +can extend to other languages and to play with other things. +

+ +
+ + +
The basic language
+ + +
+ +

This tutorial will be illustrated with a toy language that we'll call +"Kaleidoscope". +Kaleidoscope is a procedural language that allows you to define functions, use +conditionals, math, etc. Over the course of the tutorial, we'll extend +Kaleidoscope to support if/then/else, operator overloading, JIT compilation with +a simple command line interface, etc.

+ +

Because we want to keep things simple, in Kaleidoscope the only datatype is a +64-bit floating point type (aka 'double' in C parlance). As such, all values +are implicitly double precision and the language doesn't require type +declarations. This gives the language a very nice and simple syntax. For +example, A simple example computes Fibonacci numbers, +which looks like this:

+ +
+
+# Compute the x'th fibonacci number.
+def fib(x)
+  if x < 3 then
+    1
+  else
+    fib(x-1)+fib(x-2)
+
+# This expression will compute the 40th number.
+fib(40)
+
+
+ +

We also allow Kaleidoscope to call into standard library functions (this LLVM +JIT makes this completely trivial). This means that you can use the 'extern' +keyword to define a function before you use it (this is also useful for mutually +recursive functions). For example:

+ +
+
+extern sin(arg);
+extern cos(arg);
+extern atan2(arg1 arg2);
+
+atan2(sin(.4), cos(42))
+
+
+ +

In the first incarnation of the language, we will only support basic +arithmetic: if/then/else will be added in a future installment. Another +interesting aspect of the first implementation is that it is a completely +functional language, which does not allow you to have side-effects etc. We will +eventually add side effects for those who prefer them.

+ +

In order to make this tutorial +maximally understandable and hackable, we choose to implement everything in C++ +instead of using lexer and parser generators. LLVM obviously works just fine +with these tools, and choice of these tools doesn't impact overall design.

+ +

A note about this tutorial: we expect you to extend the language and play +with it on your own. Take the code and go crazy hacking away at it. It can be +a lot of fun to play with languages!

+ +
+ + +
The Lexer
+ + +
+ +

When it comes to implementing a language, the first thing needed is +the ability to process a text file and recognize what it says. The traditional +way to do this is to use a "lexer" (aka 'scanner') +to break the input up into "tokens". Each token returned by the lexer includes +a token code and potentially some metadata (e.g. the numeric value of a number). +First, we define the possibilities: +

+ +
+
+// The lexer returns tokens [0-255] if it is an unknown character, otherwise one
+// of these for known things.
+enum Token {
+  tok_eof = -1,
+
+  // commands
+  tok_def = -2, tok_extern = -3,
+
+  // primary
+  tok_identifier = -4, tok_number = -5,
+};
+
+static std::string IdentifierStr;  // Filled in if tok_identifier
+static double NumVal;              // Filled in if tok_number
+
+
+ +

Each token returned by our lexer will either be one of the Token enum values +or it will be an 'unknown' character like '+' which is returned as its ascii +value. If the current token is an identifier, the IdentifierStr +global variable holds the name of the identifier. If the current token is a +numeric literal (like 1.0), NumVal holds its value. Note that we use +global variables for simplicity, this is not the best choice for a real language +implementation :). +

+ +

The actual implementation of the lexer is a single function gettok. +gettok is called to return the next token from standard input. Its +definition starts as:

+ +
+
+/// gettok - Return the next token from standard input.
+static int gettok() {
+  static int LastChar = ' ';
+
+  // Skip any whitespace.
+  while (isspace(LastChar))
+    LastChar = getchar();
+
+
+ +

+gettok works by calling the C getchar() function to read +characters one at a time from standard input. It eats them as it recognizes +them and stores the last character read but not processed in LastChar. The +first thing that it has to do is ignore whitespace between tokens. This is +accomplished with the loop above.

+ +

The next thing it needs to do is recognize identifiers, and specific keywords +like "def". Kaleidoscope does this with this simple loop:

+ +
+
+  if (isalpha(LastChar)) { // identifier: [a-zA-Z][a-zA-Z0-9]*
+    IdentifierStr = LastChar;
+    while (isalnum((LastChar = getchar())))
+      IdentifierStr += LastChar;
+
+    if (IdentifierStr == "def") return tok_def;
+    if (IdentifierStr == "extern") return tok_extern;
+    return tok_identifier;
+  }
+
+
+ +

Note that it sets the 'IdentifierStr' global whenever it lexes an +identifier. Also, since language keywords are matched by the same loop, we +handle them here inline. Numeric values are similar:

+ +
+
+  if (isdigit(LastChar) || LastChar == '.') {   // Number: [0-9.]+
+    std::string NumStr;
+    do {
+      NumStr += LastChar;
+      LastChar = getchar();
+    } while (isdigit(LastChar) || LastChar == '.');
+
+    NumVal = strtod(NumStr.c_str(), 0);
+    return tok_number;
+  }
+
+
+ +

This is all pretty straight-forward code for processing input. When reading +a numeric value from input, we use the C strtod function to convert it +to a numeric value that we store in NumVal. Note that this isn't doing +sufficient error checking: it will incorrect read "1.23.45.67" and handle it as +if you typed in "1.23". Feel free to extend it :). Next we handle comments: +

+ +
+
+  if (LastChar == '#') {
+    // Comment until end of line.
+    do LastChar = getchar();
+    while (LastChar != EOF && LastChar != '\n' & LastChar != '\r');
+    
+    if (LastChar != EOF)
+      return gettok();
+  }
+
+
+ +

We handle comments by skipping to the end of the line and then returnning the +next comment. Finally, if the input doesn't match one of the above cases, it is +either an operator character like '+', the end of file. These are handled with +this code:

+ +
+
+  // Check for end of file.  Don't eat the EOF.
+  if (LastChar == EOF)
+    return tok_eof;
+  
+  // Otherwise, just return the character as its ascii value.
+  int ThisChar = LastChar;
+  LastChar = getchar();
+  return ThisChar;
+}
+
+
+ +

With this, we have the complete lexer for the basic Kaleidoscope language. +Next we'll build a simple parser that uses this to +build an Abstract Syntax Tree. If you prefer, you can jump to the main tutorial index page. +

+ +
+ + +
+
+ Valid CSS! + Valid HTML 4.01! + + Chris Lattner
+ The LLVM Compiler Infrastructure
+ Last modified: $Date: 2007-10-17 11:05:13 -0700 (Wed, 17 Oct 2007) $ +
+ + diff --git a/docs/tutorial/index.html b/docs/tutorial/index.html index 43c5dab5de2..acaee03367b 100644 --- a/docs/tutorial/index.html +++ b/docs/tutorial/index.html @@ -25,17 +25,18 @@
  • Invoking the JIT
    • -
  • Implementing a simple language with LLVM +
  • Implementing a language with LLVM: Kaleidoscope
    1. The basic language, with its lexer
    2. Implementing a Parser and AST
    3. Implementing code generation to LLVM IR
      4. +
    4. Adding JIT codegen support
    5. Extending the language: if/then/else
    6. Extending the language: operator overloading
      7. -
    7. Adding JIT codegen support
      8. +
    8. Extending the language: mutable variables
    9. Thoughts and ideas for extensions
    • - \ No newline at end of file +