--- /dev/null
+<!--\r
+ Licensed to the Apache Software Foundation (ASF) under one or more\r
+ contributor license agreements. See the NOTICE file distributed with\r
+ this work for additional information regarding copyright ownership.\r
+ The ASF licenses this file to you under the Apache License, Version 2.0\r
+ (the "License"); you may not use this file except in compliance with\r
+ the License. You may obtain a copy of the License at\r
+\r
+ http://www.apache.org/licenses/LICENSE-2.0\r
+\r
+ Unless required by applicable law or agreed to in writing, software\r
+ distributed under the License is distributed on an "AS IS" BASIS,\r
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\r
+ See the License for the specific language governing permissions and\r
+ limitations under the License.\r
+-->\r
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">\r
+<html>\r
+<head>\r
+ <meta content="text/html; charset=ISO-8859-1"\r
+ http-equiv="content-type">\r
+ <title>Derby Type System</title>\r
+</head>\r
+<body>\r
+<h2>Derby Type System</h2>\r
+The Derby type system is mainly contained in the <span\r
+ style="font-family: monospace;">org.apache.derby.iapi.types</span>\r
+package. The main two classes are <span style="font-family: monospace;">DataValueDescriptor\r
+</span>and\r
+<span style="font-family: monospace;">DataTypeDescriptor</span>.<br>\r
+<h3>DataValueDescriptor</h3>\r
+Values in Derby are always represented by instances of <span\r
+ style="font-family: monospace;">org.apache.derby.iapi.types.DataValueDescriptor</span>,\r
+which might have\r
+been better named <span style="font-family: monospace;">DataValue</span>.\r
+<span style="font-family: monospace;"><span\r
+ style="font-family: monospace;">DataValueDescriptor</span></span>, or\r
+DVD for short, is\r
+mainly used to represent SQL data values, though it is used for other\r
+internal types. <span style="font-family: monospace;">DataValueDescriptor\r
+</span>is a Java \r
+interface and in general all values are manipulated through interfaces\r
+and not the Java class implementations such as <span\r
+ style="font-family: monospace;">SQLInteger</span>.\r
+DVDs are mutable\r
+(their value can change) and can represent <span\r
+ style="font-family: monospace;">NULL </span>or a valid value. Note\r
+that SQL <span style="font-family: monospace;">NULL </span>is\r
+represented by a <span style="font-family: monospace;">DataValueDescriptor\r
+</span>with a state of <span style="font-family: monospace;">NULL</span>,\r
+not a <span style="font-family: monospace;">null </span>Java\r
+reference to a <span style="font-family: monospace;">DataValueDescriptor</span>.<br>\r
+<br>\r
+Generally the Derby\r
+engine works upon an array of DVD's that represent a row, which can\r
+correspond to a row in a table, a row in a <span\r
+ style="font-family: monospace;">ResultSet </span>to be returned to\r
+the application or an intermediate row in a query. The DVD's within\r
+this array are re-used for each row processed, this is why they are\r
+mutable. For example in reading rows from the store a single DVD\r
+is used to read a column's value for all the rows processed. This is to\r
+benefit performance, thus in a table scan of one million rows Derby\r
+does not create one million objects, which would be the case if the\r
+type system was immutable, like the Java object wrappers\r
+java.lang.Integer\r
+etc.<br>\r
+<br>\r
+The methods in <span style="font-family: monospace;">DataValueDescriptor\r
+</span>can be broken into\r
+these groups<span style="font-family: monospace;"><span\r
+ style="font-family: times new roman,times,serif;"><br>\r
+</span></span>\r
+<ul>\r
+ <li><span style="font-family: monospace;">getXXX </span>methods\r
+to help implement <span style="font-family: monospace;">java.sql.ResultSet.getXXX</span>\r
+methods. Thus a <span style="font-family: monospace;">ResultSet.getInt()</span>\r
+corresponds to the <span style="font-family: monospace;">DataValueDescriptor.getInt()</span>\r
+method.</li>\r
+ <li>setValue methods to help implement <span\r
+ style="font-family: monospace;">java.sql.PreparedStatement.setXXX</span>\r
+methods. Thus a <span style="font-family: monospace;">PreparedStatement.setInt()</span>\r
+corresponds to the <span style="font-family: monospace;">DataValueDescriptor.setValue(int)</span>\r
+method. These methods perform overflow and other range checks, e.g.\r
+setting a <span style="font-family: monospace;">long </span>into a <span\r
+ style="font-family: monospace;">SQLInteger </span>checks\r
+to see that the value is within the range of an <span\r
+ style="font-family: monospace;">int</span>, if not\r
+an exception is thrown.<br>\r
+These methods are also used to implement casts and other data type\r
+conversion, thus ensuring consistent type conversions for Derby within\r
+SQL and JDBC.<br>\r
+ </li>\r
+ <li>Methods to support SQL operators, e.g. <span\r
+ style="font-family: monospace;">isNull</span>.</li>\r
+ <li>Methods to read and write to disk, or strictly to convert the\r
+value into a byte representation, e.g.<span\r
+ style="font-family: monospace;">writeExternal</span>.<br>\r
+ </li>\r
+</ul>\r
+<span style="font-family: monospace;"><span\r
+ style="font-family: times new roman,times,serif;"></span></span>\r
+<h4>Type Specific\r
+Interfaces</h4>\r
+To support operators specific to a type, or set of types, Java\r
+interfaces that extend <span style="font-family: monospace;">DataValueDescriptor<span\r
+ style="font-family: times new roman,times,serif;"> exist:<br>\r
+</span></span>\r
+<ul>\r
+ <li><span style="font-family: monospace;">NumberDataValue </span>-\r
+Methods for operators on numeric types, such as <span\r
+ style="font-family: monospace;">INTEGER</span>, <span\r
+ style="font-family: monospace;">DECIMAL</span>, <span\r
+ style="font-family: monospace;">REAL</span>, e.g. plus for the SQL +\r
+operator.<br>\r
+ </li>\r
+ <li><span style="font-family: monospace;">StringDataValue </span>-\r
+Methods for operators on character types, such as <span\r
+ style="font-family: monospace;">CHAR</span>, <span\r
+ style="font-family: monospace;">VARCHAR</span><span\r
+ style="font-family: monospace;"></span>.</li>\r
+ <li><span style="font-family: monospace;">BitDataValue </span>-\r
+Methods for operators on binary types, such as <span\r
+ style="font-family: monospace;">CHAR FOR BIT DATA</span>, <span\r
+ style="font-family: monospace;">BLOB</span><span\r
+ style="font-family: monospace;"></span>.</li>\r
+ <li><span style="font-family: monospace;">DateTimeDataValue </span>-\r
+Methods for operators on character types, such as <span\r
+ style="font-family: monospace;">CHAR</span>, <span\r
+ style="font-family: monospace;">VARCHAR</span><span\r
+ style="font-family: monospace;"></span>.</li>\r
+ <li><span style="font-family: monospace;">BooleanDataValue </span>-\r
+Methods for operators on <span style="font-family: monospace;">BOOLEAN\r
+ </span><span style="font-family: monospace;"></span>type<span\r
+ style="font-family: monospace;"></span><span\r
+ style="font-family: monospace;"></span>.</li>\r
+</ul>\r
+<h4>Language Compilation</h4>\r
+Much of the generate code for language involves the type system. E.g.\r
+SQL operators are converted to method calls on interfaces within the\r
+type system, such as <span style="font-family: monospace;">DataValueDesciptor\r
+</span>or NumberDataValue. Thus all\r
+this generated code makes method calls through interface method calls.\r
+The language has a policy/style of generating fields with holder\r
+objects for the result of any operation. This holder\r
+<span style="font-family: monospace;">DataValueDescriptor </span>is\r
+then re-used for all the operations within that\r
+query execution, thus saving object creation when the operation is\r
+called on multiple rows. The generated code does not create the initial\r
+value for the field, instead the operator method or <span\r
+ style="font-family: monospace;">DataValueFactory\r
+</span>methods create instance the first time that the result is passed\r
+in as\r
+<span style="font-family: monospace;">null</span>. The approximate Java\r
+code for this would be (note the generator\r
+generates to byte code directly).<br>\r
+<br>\r
+<span style="font-family: monospace;"> // instance field to\r
+hold the result of the minus</span><br style="font-family: monospace;">\r
+<span style="font-family: monospace;"> private\r
+NumberDataValue f7;</span><br style="font-family: monospace;">\r
+<br style="font-family: monospace;">\r
+<span style="font-family: monospace;"> \r
+...</span><br style="font-family: monospace;">\r
+<br style="font-family: monospace;">\r
+<span style="font-family: monospace;"> // code within\r
+a generated method</span><br style="font-family: monospace;">\r
+<span style="font-family: monospace;"> f7 = value.minus(f7);</span><br\r
+ style="font-family: monospace;">\r
+<br>\r
+<h4><span style="font-family: monospace;"><span\r
+ style="font-family: times new roman,times,serif;"></span></span></h4>\r
+<h4>Interaction with\r
+Store</h4>\r
+The store knows little about how values represent themselves in bytes,\r
+all that knowledge is contained within the DVD implementation.<br>\r
+The exception is SQL <span style="font-family: monospace;">NULL </span>handling,\r
+the store handles <span style="font-family: monospace;">NULL </span>values\r
+consistently, as a null bit in the status byte for a field. Thus\r
+<span style="font-family: monospace;">readExternal </span>and <span\r
+ style="font-family: monospace;">writeExternal </span>are never called\r
+for a\r
+<span style="font-family: monospace;">DataValueDescriptor </span>that\r
+is <span style="font-family: monospace;">NULL</span>.<br>\r
+<h4>Delayed Object Creation</h4>\r
+When a value reads itself from its byte representation it is required\r
+that the least amount of work is performed to obtain a useful\r
+representation of a value. This is because the value being read from\r
+disk may never be returned to the application, or returned but never\r
+used by the application. The first case can occur when a qualification\r
+in the SQL statement is executed at the language layer and not pushed\r
+down to the store, thus the row is fetched from the store but filtered\r
+out at the language layer. Taking <span style="font-family: monospace;">SQLDecimal\r
+</span>as an example, the byte\r
+format is a representation of a <span\r
+ style="font-family: monospace;">java.math.BigInteger</span> instance\r
+along with a scale. Taking the simple approach that <span\r
+ style="font-family: monospace;">SQLDecimal </span>would\r
+always use a java.math.BigDecimal, then this is the steps that would\r
+occur when reading a <span style="font-family: monospace;">DECIMAL </span>column:<br>\r
+<ol>\r
+ <li>Read <span style="font-family: monospace;">BigInteger </span>format\r
+into byte array, read scale</li>\r
+ <li>New <span style="font-family: monospace;">BigInteger </span>instance\r
+from byte array - <span style="font-weight: bold;">2 object creations\r
+and byte array copy</span></li>\r
+ <li>New <span style="font-family: monospace;">BigDecimal </span>instance\r
+from BigInteger and scale - <span style="font-weight: bold;">1 object\r
+creation</span><br>\r
+ </li>\r
+</ol>\r
+Now think about a million row table scan with a DECIMAL column that\r
+returns 1% of the rows to the application, filtering at the language\r
+layer.<br>\r
+<br>\r
+This simple <span style="font-family: monospace;">SQLDecimal </span>implementation\r
+will create 3 million objects and\r
+do 1 million byte array copies.<br>\r
+<br>\r
+The smart (and current) implementation of <span\r
+ style="font-family: monospace;">SQLDecimal </span>will delay steps 2\r
+and 3 until there is an actual need for a <span\r
+ style="font-family: monospace;">BigDecimal </span>object, e.g when\r
+the application calls <span style="font-family: monospace;">ResultSet.getBigDecimal</span>.\r
+So assuming the\r
+application calls <span style="font-family: monospace;">getBigDecimal </span>for\r
+every row it receives, then, since\r
+only 1% of the rows are returned, 30,000 objects are created and 10,000\r
+byte copies are made, thus saving 2,970,000 object creations and\r
+990,000 byte array copies and the garbage collection overhead of those\r
+short lived objects.<br>\r
+<br>\r
+This delayed object creation increases the complexity of the\r
+<span style="font-family: monospace;">DataValueDescriptor </span>implementation,\r
+but the performance benefit is well\r
+worth it. The complexity comes from the implementation maintaining dual\r
+state, in <span style="font-family: monospace;">SQLDecimal </span>case\r
+the value is represented by either the raw\r
+value, or by a <span style="font-family: monospace;">BigDecimal </span>object.\r
+Care is taken in the implementation\r
+to always access the value through methods, and not the fields\r
+directly. String based values such as <span\r
+ style="font-family: monospace;">SQLChar </span>also perform this\r
+delayed\r
+object creation to String, as creating a String object requires two\r
+object creations and a char array copy. In the case of <span\r
+ style="font-family: monospace;">SQLChar </span>though,\r
+the raw value is maintained as a char array and not a byte array, this\r
+is because the char[] can be used as-is as the value, e.g. in string\r
+comparisons.<br>\r
+<h3>DataValueFactory</h3>\r
+Specific instances of <span style="font-family: monospace;">DataValueDescriptor\r
+</span>are mostly created through\r
+the <span style="font-family: monospace;">DataValueFactory </span>interface.\r
+This hides the implementation of types\r
+from the JDBC, language and store layers. This interface includes\r
+methods to:<br>\r
+<ul>\r
+ <li>generate new <span style="font-family: monospace;">NULL </span>values\r
+for specific SQL types.<br>\r
+ </li>\r
+ <li>generate specific types from Java primitves or Java objects (such\r
+as String). The returned type corresponds to the JDBC mapping for the\r
+Java type, e.g. <span style="font-family: monospace;">SQLInteger </span>for\r
+ <span style="font-family: monospace;">int</span>. Where the Java\r
+type can map to\r
+multiple SQL types there are specific methods such as getChar,\r
+getVarchar.</li>\r
+</ul>\r
+<h3><span style="font-family: monospace;"></span></h3>\r
+<h3>DataTypeDescriptor</h3>\r
+The SQL type of a column, value or expression is represented by an\r
+instance of <span style="font-family: monospace;">org.apache.derby.iapi.types.DataTypeDescriptor.</span><span\r
+ style="font-family: monospace;"> DataTypeDescriptor</span><span\r
+ style="font-family: monospace;"> contains three key\r
+pieces of information:<br>\r
+</span>\r
+<ol>\r
+ <li>The fundamental SQL type, e.g. <span\r
+ style="font-family: monospace;">INTEGER</span>, <span\r
+ style="font-family: monospace;">DECIMAL, represented by a\r
+org.apache.derby.iapi.types.TypeId.<br>\r
+ </span></li>\r
+ <li>Any length, precision or scale attributes, e.g. length for <span\r
+ style="font-family: monospace;">CHAR</span>, precision & scale for\r
+ <span style="font-family: monospace;">DECIMAL</span>.</li>\r
+ <li>Is the type nullable</li>\r
+</ol>\r
+Note that a <span style="font-family: monospace;">DataValueDescriptor </span>is\r
+not tied to any <span style="font-family: monospace;">DataTypeDescriptor</span>,\r
+thus setting a value into a <span style="font-family: monospace;">DataValueDescriptor\r
+</span>that does not conform to the intended <span\r
+ style="font-family: monospace;">DataTypeDescriptor </span>is allowed.\r
+The value is checked in an explict normalization phase. As an example,\r
+an application can use <span style="font-family: monospace;">setBigDecimal()</span>\r
+to set <span style="font-family: monospace;">199.0</span> to a\r
+parameter that is marked as being <span style="font-family: monospace;">DECIMAL(4,2)</span>.\r
+Only on the execute phase will the out of range exception be raised.<br>\r
+<ol>\r
+</ol>\r
+<h3>Issues</h3>\r
+<h4>Interfaces or Classes</h4>\r
+Matching the interface type hierachy is a implementation (class)\r
+hierachy complete with abstract types, for example DataType (again\r
+badly named) is the abstract root for all implementations of\r
+DataValueDescriptor, and NumberDataType for NumberDataValue. Code would\r
+be smaller and faster if the interfaces were removed and the official\r
+api became the public methods of these abstract classes. The work\r
+involved here is fixing the code generation involving types, regular\r
+java code would be compiled correctly with any change, but the\r
+generated code needs to be change by hand, to change interface calls to\r
+method calls. Any change like this should probably rename the abstract\r
+classes to short descriptive names, liker DataValue and NumberValue.<br>\r
+<h4>DataValueFactory</h4>\r
+There is demonstrated need to hide the implementation of DECIMAL as\r
+J2ME, J2SE and J2SE5 require different versions, thus a type\r
+implementation factory is required. However it seems to be too generic\r
+to have the ability to support different implementations of INTEGER,\r
+BIGINT and some other fundemental types. Thus maybe the code could be\r
+simplified to allow use of SQLInteger, SQLLong and others directly. At\r
+least the SQL types that are implemented using Java primitives.<br>\r
+<h4>Result Holder Generation</h4>\r
+The dynamic creation of result holders (see language section) means\r
+that all operators have to check for the result reference being passed\r
+in being null, and if so create a new instance of the desired type.\r
+This check seems inefficient as it will be performed once per\r
+operation, again, imagine the million row query. In addition the field\r
+that holds the result holder in the generated code is assigned each\r
+time to the same value, inefficient. It seems that the code using the\r
+type system, generated or coded, can set up the result holder at\r
+initialization time, thus removing the need for the check and field\r
+assignment, leading to faster smaller code.<br>\r
+<h4>NULL and operators</h4>\r
+The operators typically have to check for incoming NULL values and\r
+assign the result to be NULL if any of the inputs are NULL. This\r
+combined with the result holder generation issue leads to a lot of\r
+duplicate code checking to see if the inputs are NULL. It's hard to\r
+currently do this in a single method as the code needs to determine if\r
+the inputs are NULL, generate a result holder and return two values (is\r
+the result NULL and what is the result holder). Splitting the operator\r
+methods into two would help as at least the NULL checks could be in the\r
+super-class for all the types, rather than in each implementation. In\r
+addition this would lead to the ability to generate to a more efficient\r
+operator if the inputs are not nullable. E.g for the + operator there\r
+could be <span style="font-family: monospace;">plus()</span> and <span\r
+ style="font-family: monospace;">plusNotNull()</span> methods, the <span\r
+ style="font-family: monospace;">plus()</span> being implemented in the\r
+NumberDataType class, handling NULL inputs and calling plusNotNull(),\r
+with the plusNotNull() implemented in the specific type.<br>\r
+<h4>Operators and self</h4>\r
+It seems the operator methods should almost always be acting on thier\r
+own value, e.g. the plus() method should only take one input and the\r
+result is the value of the receiver (self) added to the input.\r
+Currently the plus takes two inputs and probably in most if not all\r
+cases the left input is the receiver. The result would be smaller code\r
+and possible faster, as the method calls on self would not be through\r
+an interface.<br>\r
+</body>\r
+</html>\r
projects / Benchmarks_CSolver.git / blobdiff