\r
\section{\textbf{Introduction}}\r
\r
-\section{\textbf{Approach}}\r
+\section{\textbf{Device Approach}}\r
\r
\subsection{\textbf{Records}}\r
Each record has the following information included in it:\r
\begin{itemize}\r
\item Machine ID of the device creating the record\r
\item The vector clock using the largest clock values from each device it knows and its own largest clock value incremented by 1.\r
- \item Add a random salt (or nonce) for the encryption safety\r
\item Data payload\r
\item HMAC of the record.\r
\end{itemize}\r
\r
-\r
+Records can be identified by the machine ID and the local machine clock, hereby referred to as the record ID.\r
\r
\subsubsection{\textbf{Types of Payloads}}\r
The different types of record payloads are:\r
\begin{itemize}\r
- \item Delete notifications\r
+\r
+ \item Transactions\r
\begin{itemize}\r
- \item Contains the HMAC of records that were deleted by devices, Their vector clocks and the server sequence numbers.\r
- \item Generated when a device deletes a record.\r
- \item The delete notification with the largest server sequence number in its delete payload is the live one (the one that contains the largest server sequence number of the record deleted to date). \r
+ \item Contains:\r
+ \begin{itemize}\r
+ \item Transaction ID\r
+ \item key-value pair gets (reads)\r
+ \item A guard condition (boolean condition) that can be evaluated on the key-value gets.\r
+ \item A set of key-value pairs that are to be updated if the guard condition is met.\r
+ \item Can only get and set key-value pairs that are from 1 arbitrator. Getting and/or setting key-value pairs from more than 1 arbitrator causes the transaction to be invalid and dead.\r
+ \end{itemize}\r
\end{itemize}\r
\item Commit notifications\r
\begin{itemize}\r
- \item Contains list of transactions that are committed in order of commit and the current key-value pair for that key.\r
- \item Generated by the arbitrator of a key and only the for that key (1 arbitrator per key).\r
- \item Used in the case that there is a key value pair that needs reordering.\r
+ \item Contains the commit of a single transaction, the whole transaction.\r
+ \item There is 1 commit per transaction.\r
+ \item Generated by the arbitrator for the set of key-value gets and sets in the transaction.\r
\end{itemize}\r
\item Abort notifications\r
\begin{itemize}\r
\item Contains a transaction ID of an aborted transaction and the machine ID of the device that created that transaction.\r
- \item Causes a transaction to be aborted.\r
- \end{itemize}\r
- \item Abort acknowledgement notifications\r
- \begin{itemize}\r
- \item Contains a transaction ID of an aborted transaction, the machine ID of the device that created that transaction and the abort notification ID that this is acknowledging.\r
- \item Causes an abort notification to become dead.\r
- \item Is generated by the device that had created an aborted transaction as an acknowledgement that it saw the aborted transaction notification.\r
- \item This payload type immediately becomes dead (not live) upon insertion into the data structure.\r
+ \item Causes a transaction to be aborted, key-values not used in updates.\r
\end{itemize}\r
\item Data structure re-size notifications\r
\begin{itemize}\r
\item Contains new size of data structure (number of record allowed in the data structure or something like that).\r
- \item Causes old data Structure re-size notification to no longer be live.\r
\end{itemize}\r
- \item Server sequence number for a specific record notifications\r
+ \item Server sequence number confirmations.\r
\begin{itemize}\r
- \item Contains a record HMAC and the server sequence number for that record\r
+ \item Contains a record ID and the server sequence number for that record that the server reported.\r
+ \item Created by any device if that device finds a record with a server sequence number that does not have a server sequence number conformation yet.\r
\end{itemize}\r
- \item Transactions\r
+ \item Delete notifications\r
\begin{itemize}\r
- \item Contains:\r
- \begin{itemize}\r
- \item Transaction ID\r
- \item A guard condition that can be evaluated\r
- \item A set of key-value pairs that are to be updated if the guard condition is met.\r
- \end{itemize}\r
+ \item Contain the server sequence number of the record that was deleted.\r
+ \item Generated when a device deletes a record.\r
\end{itemize}\r
\end{itemize}\r
\r
+\subsection{\textbf{Pulling the data structure}}\r
+To pull the latest version of the data structure the following is done:\r
+\begin{enumerate}\r
+ \item Make a pull request to the server and get all the records sent back.\r
+ \item Separate the records by machine ID.\r
+ \item For each machine ID, order the records based on that machine IDs clock within each of the records.\r
+ \item Check the data structure for any malicious activity (discussed below)\r
+\end{enumerate}\r
\r
-\subsection{\textbf{Updates (Online Updates)}}\r
+\subsection{\textbf{Updates}}\r
Updates take place as follows:\r
\begin{enumerate}\r
\item A device pulls the latest version of the data structure. If the device cannot pull the latest version because of network connectivity or some other issues then that device will just work using the local copy of the data structure it has.\r
+ \item Check the pulled data structure for any malicious activity (as stated in a section below) if not done already.\r
+ \item Check if any records in the current server need to be deleted (have delete notifications in data structure but the delete never took place), if so then delete them.\r
+ \item Check that the size of the data structure will not exceed the max size when the new record is inserted. If it does then prepare to delete records by inserting delete payloads in the new record (discussed below).\r
\item The device makes a record as follows:\r
\begin{enumerate}\r
\item Adds its machine ID.\r
\item Creates a vector clock using the largest clock values from each device it knows and its own largest clock value incremented by 1.\r
- \item Add a random salt (or nonce) for the encryption safety\r
- \item Fill the record data section with the transactions, key-value pairs, ext.\r
- \item Fill the remainder of the data section with rescued key-value pairs, transactions, ext (Discussed later).\r
+ \item Fill the record payload section with the transactions and other types of payloads.\r
+ \item Fill the empty space of the record payload with server sequence number confirmations for records that have yet to have their server sequence numbers confirmed.\r
+ \item Fill the empty space of the record payload with rescued key-value pairs, transactions, ext (Discussed later).\r
\item Pad the record to be the same size for all records.\r
\item Calculate the HMAC of the record and add that to the record\r
\item Encrypt the record.\r
\end{enumerate}\r
\item Send the record to the server for insertion into the device's queue.\r
- \item Wait for response from server stating the new records (the one just sent) server sequence number. Save this server sequence number for when creating the next record.\r
- \item \r
-\end{enumerate}\r
-\r
-\r
-\subsection{\textbf{Updates when offline}}\r
-When offline and making updates, the devices should use their local copy of the data structure but do no deletes. When connection is reestablished the following should take place:\r
-\r
-\begin{enumerate}\r
- \item Pull the latest version of the data structure.\r
- \item Update local copy of the data structure except for own devices device queue (do deletions as needed)\r
- \item Calculate many records are "new" to the data structure and pick the same amount to be deleted\r
- \item Push the updates and the deletes to the server\r
- \item Wait for sequence numbers for the recently pushed records\r
- \item Push the sequence numbers for the recently pushed records (using online updates from the section above)\r
+ \item Issue any server commands such as deletes to the server.\r
\end{enumerate}\r
\r
-This kind of update will result in the latest key-value pair being the last pushed record from this update (if no other updates are occurring at the same time). The arbitrator can then commit or abort as needed but in the mean time the key-value pair may be an old one (but have the largest server sequence number).\r
+If there is a connectivity issue then all the records will be held by the local device until connection is resumed then pushed to the server in the order which they occurred. Also the device can only delete records for which there is a server sequence number. At some point the device could run out of records to delete (it hasn't connected to the server in a while) in which case the device will not be able to delete any records.\r
\r
\subsection{\textbf{Deletions}}\r
When deciding which records to delete the following is to be done:\r
\begin{enumerate}\r
\item Order all the records in order based on their server sequence numbers\r
- \item Calculate the difference between the current size of the data structure and the minimum size of the data structure (lets call this $m$)\r
+ \item Calculate the difference between the current size of the data structure and the minimum size of the data structure (lets call this $m$). Note: count newly inserted records towards the total size of the data structure if doing deletes while doing updates.\r
\item Delete the oldest m records based on the ordering from step 1. \r
\begin{itemize}\r
- \item If a record to be deleted has live data in it then the whole data structure needs to be resized.\r
+ \item If a record to be deleted has live data in it then the whole data structure needs to be re-sized.\r
\end{itemize}\r
\end{enumerate}\r
\r
-Note this makes that size of the data structure be bounded.\r
+Note this makes that size of the data structure be bounded:\r
If there are $n$ devices and the data structure has a minimum size of $m$. Then the max size of the data structure is given by $m + n -1$ for the case when all the devices make an update at the same time. \r
\r
\subsection{\textbf{Rescuing Transactions, Commits, Aborts, Ext}}\r
-Data should be proactively rescued from the "oldest" records currently in the data structure. Unused space in new records should be used to rescue data from old records so that when it comes time to delete the old records, there are no live pieces of data that need to be rescued. When a piece of data is rescued, it is rescued with its vector clock as well (so that the time of that data can be saved).\r
+Data should be proactively rescued from the "oldest" records currently in the data structure. Unused space in new records should be used to rescue data from old records so that when it comes time to delete the old records, there are no live pieces of data that need to be rescued. When a piece of data is rescued, it is rescued with its vector clock as well (so that the time of that data can be saved).\\\r
+\r
+When rescuing transactions and commits: only keep the key-value pair sets that do not have a newer key-value pair set (no other transaction/commits sets that key-value pair later in the future). This means that transactions/commits can shrink in size.\r
\r
When deciding which data to rescue the following is to be done:\r
\begin{enumerate}\r
\item Order all the records in order based on their server sequence numbers\r
- \item Create an ordered list of currently live transactions, commits, aborts, ext from the oldest $n$ records from step one where the order is based on the age of the data (how old the record .\r
- \item Randomly select from the list of live data to save. Save as much as can fit in the current new record. The random selection could give higher probability to data from records that are to be deleted sooner.\r
+ \item Create an ordered list of currently live transactions, commits, aborts, ext from the oldest $n$ records from step one where the order is based on the age of the data (how old the record is).\r
+ \item Randomly select from the list of live transactions, commits, aborts, ext to save. Save as much as can fit in the current new record. The random selection could give higher probability to transactions, commits, aborts, ext from records that are to be deleted sooner.\r
\end{enumerate}\r
\r
-If a record needs to be deleted but still contains live data then the data structure needs to be resized. \r
-\r
\subsection{\textbf{Checking the Data Structure}}\r
Checking the data structure for consistency is done as follows:\r
\begin{enumerate}\r
\item Verify that each record in the data structure has an HMAC that matches the data in the record.\r
- \item Verify that there are at least as many records in the data structure as stated in the largest data structure size record.\r
- \item Make sure that for each device queue the difference between the vector clock value of the device queues clock is at most 1 between 2 consecutive messages.\r
+ \item Verify that the oldest record the server sent has a server sequence number that is equal to or less than the server sequence number in the most recent delete notification (currently live delete notification) + 1.\r
+ \item Make sure that for each device queue the difference between the vector clock value of the device queues clock is at most 1 between 2 consecutive messages for all records with server sequence numbers above the last deleted records server sequence numbers.\r
\item Verify that no currently live data Structure re-size notification is smaller than the last known data structure size. Data structure can only grow in size.\r
- \item Verify that all the server sequence numbers for the records that are currently present have unique numbers that have a difference of 1 (no gaps).\r
+ \item Verify that all the server sequence numbers for the records that are currently present have unique numbers.\r
+ \item Verify that all the server sequence numbers for the records have a difference of 1 (no gaps) for all records with server sequence numbers above the last deleted records server sequence numbers.\r
\item Verify record server sequence numbers against the stated server sequence numbers in the server sequence number notification payloads (make sure the server is not changing the sequence number on the fly).\r
- \item Verify that no to records have the same server sequence number\r
\end{enumerate}\r
\r
-\r
\subsection{\textbf{The Arbitrator}}\r
The arbitrator can:\r
\begin{enumerate}\r
\subsubsection{\textbf{Commits}}\r
Commits have the following properties\r
\begin{itemize}\r
- \item Agree with the ordering of the server sequence numbers\r
- \item Once a key-value pair is commited it can not be commited again.\r
+ \item Agree with the ordering of the server sequence numbers most of the time.\r
\item Cannot commit an already aborted transaction.\r
\item Commits state the ordering of key-value pairs.\r
- \item Can disagree with the ordering of server sequence numbers but\r
+ \item Can disagree with the ordering of server sequence numbers if arbitrator decides to do so.\r
\item Should occur frequently as to make sure that the commit order matches the server sequence ordering as closely as possible (prevent large divergence of the 2 orderings)\r
\end{itemize}\r
\r
\subsubsection{\textbf{Aborts}}\r
-Aborts are used to show which transactions have been aborted and will not be used in the total ordering of the transactions. Aborts are considered live until an abort acknowledgement is presented.\r
-When the transaction is aborted then the devices should simply act as if it were never present when evaluating for the latest key-value pair.\r
+\r
+\begin{itemize}\r
+ \item Aborts are used to show which transactions have been aborted based on the arbitrators decision.\r
+ \item Aborts are considered live until an abort acknowledgement is presented.\r
+ \r
+\end{itemize}\r
+ \r
\r
\subsection{\textbf{Live Status}}\r
Live Status of entries:\r
\begin{enumerate}\r
- \item Key-Value Entry/Data Entry is dead if either:\r
- \begin{enumerate}\r
- \item There is a newer key-value pair:\r
+\r
+ \item Delete notifications\r
+ \begin{itemize}\r
+ \item Live if it deleted the largest known server sequence number to be deleted (most recent delete).\r
+ \end{itemize}\r
+ \r
+ \item Commit notifications\r
+ \begin{itemize}\r
+ \item Live until all the key-value pair sets in the transaction commit are dead.\r
+ \begin{itemize}\r
+ \item key-value pairs are dead when a commit for a transaction that sets the same key-value pair occurs with a larger vector clock.\r
+ \end{itemize}\r
+ \end{itemize}\r
+ \r
+ \item Abort notifications\r
+ \begin{itemize}\r
+ \item Live until the device whos machine ID is in the abort notification makes an update to the data structure that contains a vector clock that is more in the future than the vector clock for this abort notification.\r
+ \end{itemize}\r
+ \r
+ \item Data structure re-size notifications\r
\begin{itemize}\r
- \item There is a transaction with a newer vector clock value.\r
- \item There is a commit that for this key-value pair.\r
- \item There is an abort for this key-value pair.\r
+ \item Live if it contains the largest target size of the data structure.\r
\end{itemize}\r
- \item It is incomplete.\r
- \item It is an abort notification that has an abort notification acknowledgment\r
- \item It is an abort notification acknowledgment (dead on arrival).\r
- \end{enumerate}\r
- \item Data is live if there are multiple versions of the same data (key-value pair) in which the vector clock values show concurrency. All versions are kept live the arbitrator arbitrates.\r
- \item Multiple versions of the same data (same transaction ID for example) are not all live. Only the version with the largest server sequence number is live. \r
\r
+ \item Server sequence number confirmations.\r
+ \begin{itemize}\r
+ \item Live until the record that this notification is reporting on is deleted.\r
+ \end{itemize}\r
+ \r
+ \item Transactions\r
+ \begin{itemize}\r
+ \item Is dead if it is invalid (contains keys-values for multiple arbitrators)\r
+ \item Live until a commit or abort notification for this transaction is generated.\r
+ \end{itemize}\r
+ \r
+\end{enumerate}\r
+\r
+\r
+\section{\textbf{Server Approach}}\r
+\r
+The servers view of the system is in terms of data slots where each data slot holds a single record, has a monotonically increasing number associated with it (server sequence number) for the record that currently is present in that data slot and can be set or deleted. A server may have a finite amount of memory which it can partition into slots, reusing memory that newly deleted slots used to occupy.\r
+\r
+There are 3 types of requests from a device that the server must respond to:\r
+\begin{enumerate}\r
+ \item Pull all current slots.\r
+ \item Put a new record in a slot.\r
+ \item Delete a slot.\r
+\end{enumerate}\r
+\r
+\subsection{\textbf{Pull all current slots}}\r
+In this case the server will simply send back all active slots (slots that have data) in any order along with each slots server sequence number. It is the job of the devices to order the slots.\r
+\r
+\subsection{\textbf{Put a new record in a slot}}\r
+In this case the server will:\r
+\begin{enumerate}\r
+ \item Receive a record data from a device.\r
+ \item Put this record data in an empty slot.\r
+ \item Assign the just received record the next number in the server sequence numbers.\r
\end{enumerate}\r
+If more than 1 put request is made at the same time, the server is free to order the requests however it wishes.\r
\r
+\subsection{\textbf{Delete a slot}}\r
+In this case the server will delete the data in the slot that has the server sequence number that matches the server sequence number in the delete request. The server could delay the delete if it wishes (if it has plenty of space for new slots).\r
\r
\section{\textbf{System Guarantees}}\r
\begin{itemize}\r
\item Devices can operate offline and re-sync with the system and get a consistent view of the system\r
\item If the server tries to hold a device on an older version of the data structure, that device can eventually rejoin the main data structure without problems.\r
\item Devices that have a transaction aborted will be able to be notified about the abort indefinitely (no time frame when notification must be accepted).\r
- \items Server cannot hold a device on an old version of the data structure and then move them to a newer version of the data structure without being detected (The server sequence numbers would reveal conflicts or gaps or both).\r
+ \item Server cannot hold a device on an old version of the data structure and then move them to a newer version of the data structure without being detected (The server sequence numbers would reveal conflicts or gaps or both).\r
\r
\end{itemize}\r
\r
\r
+\section{System Correctness}\r
+\r
\end{document}
\ No newline at end of file
projects / iotcloud.git / commitdiff