Execute shared library. Do away with LLVM JIT

.travis.yml

@@ -20,6 +20,7 @@ addons:
     - libjsoncpp-dev
     - libsecp256k1-dev
     - llvm-10-dev
+    - clang-10
 cache:
   directories:
   - $HOME/openssl

CMakeLists.txt

@@ -1,6 +1,6 @@
 cmake_minimum_required(VERSION 3.5.1)
 
-project(ScillaVM)
+project(ScillaRTL)
 
 # detect operating system
 message(STATUS "We are on a ${CMAKE_SYSTEM_NAME} system")
@@ -53,8 +53,7 @@ set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
 # C++ standard
 set(CMAKE_CXX_STANDARD 14)
 
-add_subdirectory(libjitd)
-add_subdirectory(libsrtl)
+add_subdirectory(libScillaRTL)
 add_subdirectory(runners)
 add_subdirectory(testsuite)
 add_subdirectory(deps/cryptoutils EXCLUDE_FROM_ALL)
@@ -67,14 +66,12 @@ set_target_properties(expr-runner
     PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
 set_target_properties(ScillaRTL
     PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib)
-set_target_properties(ScillaJITD
-    PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib)
 install(
     TARGETS scilla-runner expr-runner
     DESTINATION ${CMAKE_INSTALL_PREFIX}/bin
 )
 install(
-    TARGETS ScillaRTL ScillaJITD CryptoUtils Schnorr ff
+    TARGETS ScillaRTL CryptoUtils Schnorr ff
     DESTINATION ${CMAKE_INSTALL_PREFIX}/lib
 )
 install(

README.md

@@ -1,47 +1,31 @@
-# Scilla Virtual Machine
+# Scilla Runtime Library
 
 [![Build Status](https://travis-ci.com/Zilliqa/scilla-vm.svg?branch=master)](https://travis-ci.com/Zilliqa/scilla-vm)
 [![License](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://github.com/Zilliqa/scilla/blob/master/LICENSE)
 
-A compiled execution backend for Scilla.
+The Scilla Runtime Library provides two main functionalities
 
-The compiled execution backend is divided into two parts:
-  1. The JIT driver (JITD): JIT compiles Scilla contracts translated to
-    LLVM-IR (see [scilla-compiler](https://github.com/Zilliqa/scilla-compiler)),
-    links it with the run-time library and executes transitions.
-    compiling it to machine code, and finally executing it.
-  2. Scilla run-time library (SRTL), which the contract code interacts
-    with for performing pre-compiled operations, such as interacting with
-    the blockchain, executing builtins etc.
-
-Design docs for the [VM architecture](https://github.com/Zilliqa/scilla-backend/wiki/Scilla-Backend-Design)
-and the planned [interaction with the blockchain](https://github.com/Zilliqa/scilla-backend/wiki/Interaction-of-State-Variables-with-Blockchain)
-are available.
+  1. An entry point to execute Scilla contracts. The contract to be executed
+    must be compiled by the [Scilla LLVM compiler](https://github.com/Zilliqa/scilla-compiler),
+    and linked into a shared library object.
+  2. A collection of functions that implement various common (i.e., not specific to a contract)
+    Scilla operations and enable the compiled binary to interact with the blockchain during
+    execution.
 
 ## Build and install
 
-Add Ubuntu repository for LLVM-10. Below command line is for Ubuntu 18.04, change
-suitably for your OS (see the LLVM [apt page](https://apt.llvm.org/https://apt.llvm.org/))
-
-  ```
-  sudo add-apt-repository 'deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-10 main' -y
-  wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add -
-```
-
 Install these Ubuntu packages
-  - `sudo apt-get install build-essential cmake llvm-10-dev libboost-dev libboost-test-dev libjsoncpp-dev libboost-filesystem-dev libboost-program-options-dev libsecp256k1-dev`
+  - `sudo apt-get install build-essential clang-10 cmake libboost-dev libboost-test-dev libjsoncpp-dev libboost-filesystem-dev libboost-program-options-dev libsecp256k1-dev`
+
+Use the LLVM apt repository if clang-10 is not in your OS repository.
 
-We suggest building ScillaVM in a directory that is *not* the source directory.
+We suggest building ScillaRTL in a directory that is *not* the source directory.
   * `$git clone --recurse-submodules https://github.com/Zilliqa/scilla-vm.git`
   * `$cd scilla-vm; mkdir build; cd build`
   * `$cmake ..` configures the project.
     Additional (optional) flags:
     - `-DCMAKE_INSTALL_PREFIX=/where/to/install/scilla-vm`: To specify an install directory other
     than the default. The default installation path typically requires root permissions.
-    - `-DLLVM_DIR=/path/to/llvm/install/lib/cmake/llvm`:
-    If your LLVM is not installed in a default (system) directory,
-    provide this [flag](https://llvm.org/docs/CMake.html#embedding-llvm-in-your-project)
-    to enable `CMake` to find LLVM. Note: LLVM must be built with [RTTI support](https://llvm.org/docs/CMake.html).
     - `-DCMAKE_BUILD_TYPE=[Debug|Release|RelWithDebInfo|MinSizeRel]`: The default build is `Debug`.
   * `$make` builds the entire project. You can find the built files in `bin/` and `lib/`.
   * `$make install` installs the project.
@@ -50,8 +34,8 @@ We suggest building ScillaVM in a directory that is *not* the source directory.
   directory.
 
 ## Run
-While Scilla VM is intended to be primarily used as a library from the Zilliqa blockchain,
-we provide two wrapper executables for development, debugging and simply trying out the VM.
+While ScillaRTL is intended to be primarily used as a library from the Zilliqa blockchain,
+we provide two wrapper executables for development, debugging and simply trying it out.
 `expr-runner` takes as input, a compiled pure Scilla expression and executes it, by calling
 a wrapper function `scilla_main` generated by the compiler. If the expression is printable,
 the compiler also generates a call to print it.
@@ -61,9 +45,9 @@ With these inputs, the transition specified in the message JSON is executed. The
 is printed.
 
 ```
-$build/bin/expr-runner testsuite/expr/lit-pair-list-int.ll
+$build/bin/expr-runner -g 1000 testsuite/expr/lit-pair-list-int.ll
 
-$build/bin/scilla-runner --input-contract testsuite/contr/simple-map.ll --message testsuite/contr/simple-map.message_Increment.json --contract-info testsuite/contr/simple-map.contrinfo.json --state testsuite/contr/simple-map.state_00.json --init testsuite/contr/empty_init.json
+$build/bin/scilla-runner -g 10000 --input-contract testsuite/contr/simple-map.ll --message testsuite/contr/simple-map.message_Increment.json --contract-info testsuite/contr/simple-map.contrinfo.json --state testsuite/contr/simple-map.state_00.json --init testsuite/contr/empty_init.json
 ```
 
 ## Developer Notes
@@ -142,5 +126,8 @@ We maintain the following conventions:
 
 #### Contract execution tests
 The directory `testsuite/contr` contains text LLVM-IR files, all generated by `scilla-compiler`.
-The directory also contains the supporting JSONs required for executing transitions in these
-compiled contracts.
+The directory also contains the supporting JSONs required for executing transitions in these compiled contracts. Each LLVM-IR file `foo.ll` also has `foo.dbg.ll` that is
+a compiled version of the same contract with `debuginfo`. These LLVM-IR files can be
+updated from the Scilla LLVM compiler using the `scripts/update_contrs.sh`. This
+script updates both the debug and non-debug LLVM-IR modules and also the contract info
+for that contract.

include/ScillaVM/Debug.h → include/ScillaRTL/DLog.h

@@ -15,39 +15,44 @@
  * along with this program.  If not, see <https://www.gnu.org/licenses/>.
  */
 
-// This debug log infrastructure is inspired by:
+// This DLOG log infrastructure is inspired by:
 // https://llvm.org/docs/ProgrammersManual.html#fine-grained-debug-info-with-debug-type-and-the-debug-only-option
 
 // Provide
-//   SVM_DEBUG(dbgs() << "debug message") and
-//   SVM_DEBUG_TYPE("foo", dbgs() << "debug message")
-// in debug builds. For the former to work, DEBUG_TYPE must be
-// defined. SVM_DEBUG is equal to SVM_DEBUG_TYPE(DEBUG_TYPE, ...).
+//   SRTL_DLOG(dlog() << "debug log message") and
+//   SRTL_DLOG_TYPE("foo", dlog() << "debug log message")
+// in DLog builds. For the former to work, DLOG_TYPE must be
+// defined. SRTL_DLOG is equal to SRTL_DLOG_TYPE(DLOG_TYPE, ...).
+
+#pragma once
 
 #include <iostream>
 #include <string>
 
-namespace ScillaVM {
-bool isInCurrentDebugTypes(std::string TYPE);
-void addToCurrentDebugTypes(std::string TYPE);
-void enableAllDebugTypes();
-} // namespace ScillaVM
+namespace ScillaRTL {
+bool isInCurrentDLogTypes(std::string TYPE);
+void addToCurrentDLogTypes(std::string TYPE);
+void enableAllDLogTypes();
+} // namespace ScillaRTL
 
 #ifndef NDEBUG
 
-#define SVM_DEBUG_TYPE(TYPE, X)                                                \
+#define SRTL_DLOG_TYPE(TYPE, X)                                                 \
   do {                                                                         \
-    if (ScillaVM::isInCurrentDebugTypes(TYPE)) {                               \
+    if (ScillaRTL::isInCurrentDLogTypes(TYPE)) {                               \
       X;                                                                       \
     }                                                                          \
   } while (false)
 
-#define SVM_DEBUG(X) SVM_DEBUG_TYPE(DEBUG_TYPE, X)
+#define SRTL_DLOG(X) SRTL_DLOG_TYPE(DLOG_TYPE, X)
 
-namespace ScillaVM {
-std::ostream &dbgs();
+namespace ScillaRTL {
+std::ostream &dlogImpl(const std::string &File, int Line);
 }
+
+#define dlog() dlogImpl(__FILE__, __LINE__)
+
 #else
-#define SVM_DEBUG_TYPE(TYPE, X)
-#define SVM_DEBUG(X)
-#endif
+#define SRTL_DLOG_TYPE(TYPE, X)
+#define SRTL_DLOG(X)
+#endif // NDEBUG

include/ScillaVM/Errors.h → include/ScillaRTL/Errors.h

@@ -15,9 +15,11 @@
  * along with this program.  If not, see <https://www.gnu.org/licenses/>.
  */
 
+#pragma once
+
 #include <string>
 
-namespace ScillaVM {
+namespace ScillaRTL {
 
 struct SourceLoc {
   std::string File;
@@ -36,7 +38,7 @@ struct SourceLoc {
 struct ScillaError {
   // Error message
   std::string Msg;
-  // Where in ScillaVM was this error thrown from.
+  // Where in ScillaRTL was this error thrown from.
   SourceLoc ThrowLoc;
   // Where in the Scilla source were we when the error occurred.
   SourceLoc ScillaSrcLoc;
@@ -54,7 +56,7 @@ struct ScillaError {
   }
 };
 
-} // namespace ScillaVM
+} // namespace ScillaRTL
 
 // Throws a ScillaError object, attaching the location of error
 // Uses a default (empty) Scilla source location.

include/ScillaRTL/ScillaExec.h

@@ -0,0 +1,123 @@
+/*
+ * Copyright (C) 2019 Zilliqa
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <https://www.gnu.org/licenses/>.
+ */
+
+#pragma once
+
+#include <functional>
+#include <memory>
+#include <string>
+#include <unordered_map>
+
+#include <boost/any.hpp>
+#include <jsoncpp/json/json.h>
+
+// Forward declarations.
+namespace ScillaRTL {
+
+// Information that Scilla will need to execute contracts.
+struct ScillaParams {
+  struct StateQuery {
+    std::string Name;
+    int MapDepth;
+    std::vector<std::string> Indices;
+    bool IgnoreVal;
+  };
+  // A Scilla state contains either std::string or a MapValueT
+  // We use boost::any to capture this. Using std::variant is
+  // cumbersome because of the recursive type definition required.
+  // https://stackoverflow.com/questions/43309468/recursive-data-structure-with-variant
+  using MapValueT = std::unordered_map<std::string, boost::any>;
+
+  using FetchState_Type = std::function<bool(const StateQuery &Query,
+                                             boost::any &RetVal, bool &Found)>;
+  using FetchRemoteState_Type =
+      std::function<bool(const std::string &Addr, const StateQuery &Query,
+                         boost::any &RetVal, bool &Found, std::string &Type)>;
+  using UpdateState_Type =
+      std::function<bool(const StateQuery &Query, const boost::any &Val)>;
+
+  FetchState_Type fetchStateValue;
+  FetchRemoteState_Type fetchRemoteStateValue;
+  UpdateState_Type updateStateValue;
+
+  ScillaParams()
+      : fetchStateValue(nullptr), fetchRemoteStateValue(nullptr),
+        updateStateValue(nullptr){};
+  ScillaParams(FetchState_Type FS, FetchRemoteState_Type FRS,
+               UpdateState_Type US)
+      : fetchStateValue(FS), fetchRemoteStateValue(FRS), updateStateValue(US){};
+};
+
+class ScillaExecImpl;
+
+// Typical usage:
+//   1. ScillaContrExec SJ(...);
+//   2. (a) Deployment (b) Transition execution
+//      a. auto Output = SJ.deploy(...);
+//        OR
+//      b. auto Output = SJ.execMsg(...);
+//   3. If ScillaError exception, check remaining gas with SJ.getGasRem().
+class ScillaContrExec {
+private:
+  std::unique_ptr<ScillaExecImpl> PImpl;
+
+public:
+  // @ContrBin is the path to a contract's shared object `foo.so`
+  // generated by compiling `foo.scilla` with scilla-llvm into `foo.bc`
+  // and compiling this LLVM bitcode with `clang -shared foo.bc -o foo.so`.
+  ScillaContrExec(const ScillaParams &SPs, const std::string &ContrBin);
+  ~ScillaContrExec();
+
+  // Execute a message.
+  Json::Value execMsg(const std::string &Balance, uint64_t GasLimit,
+                      const Json::Value &InitJ, const Json::Value &Msg);
+
+  // Initialize the contract state to field initialization values in the source.
+  // This is to be called only during deployment of the contract. Never again.
+  Json::Value deploy(const Json::Value &InitJ, uint64_t GasLimit);
+
+  // What's the gas remaining from previous execution (deploy / execMsg).
+  // Useful if execution was interrupted due to an exception.
+  // Use with care if you don't want to end up with a stale value.
+  uint64_t getGasRem() const;
+};
+
+// Typical usage:
+//   1. ScillaExprExec SJ(...);
+//   2. auto Output = SJ.exec();
+//   3. If ScillaError exception, check remaining gas with SJ.getGasRem().
+class ScillaExprExec {
+private:
+  std::unique_ptr<ScillaExecImpl> PImpl;
+
+public:
+  // @ExprBin is the path to a pure scilla expressions's shared object `foo.so`
+  // generated by compiling `foo.scilexp` with expr-llvm into `foo.bc`
+  // and compiling this LLVM bitcode with `clang -shared foo.bc -o foo.so`.
+  ScillaExprExec(const ScillaParams &SPs, const std::string &ExprBin);
+  ~ScillaExprExec();
+
+  // Execute the expression.
+  std::string exec(uint64_t GasLimit);
+
+  // What's the gas remaining from previous execution of exec().
+  // Useful if execution was interrupted due to an exception.
+  // Use with care if you don't want to end up with a stale value.
+  uint64_t getGasRem() const;
+};
+
+} // namespace ScillaRTL