Skip to content

Commit

Permalink
Merge pull request #116 from C0nsultant/topic/howto_backend
Browse files Browse the repository at this point in the history 
HowTo on creating a backend
  • Loading branch information
@C0nsultant
C0nsultant authored Apr 11, 2018
2 parents 03fcad7 + abefc3c commit aea4659
Show file tree
Hide file tree
Showing 13 changed files with 833 additions and 492 deletions.
41 changes: 25 additions & 16 deletions CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ set(CORE_SOURCE
src/backend/Writable.cpp)
set(IO_SOURCE
src/IO/AbstractIOHandler.cpp
src/IO/IOTask.cpp
src/IO/AbstractIOHandlerImpl.cpp
src/IO/ADIOS/ADIOS1IOHandler.cpp
src/IO/ADIOS/ParallelADIOS1IOHandler.cpp
src/IO/ADIOS/ADIOS2IOHandler.cpp
Expand Down Expand Up @@ -507,21 +507,30 @@ endif()
# Examples ####################################################################
#

# TODO: C++ Examples
#if(EXISTS "${openPMD_BINARY_DIR}/samples/git-sample/")
# foreach(examplename ${openPMD_EXAMPLE_NAMES})
# add_test(NAME Example.py.${examplename}
# COMMAND ${examplename}
# WORKING_DIRECTORY ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}
# )
# endforeach()
#else()
# message(STATUS "Missing samples/git-sample/ directory! "
# "Skipping C++ example test!\n\n"
# "Note: run\n"
# " . ${openPMD_SOURCE_DIR}/.travis/download_samples.sh\n"
# "to add example files!")
#endif()
# C++ Examples
if(EXISTS "${openPMD_BINARY_DIR}/samples/git-sample/")
foreach(examplename ${openPMD_EXAMPLE_NAMES})
if(${examplename} MATCHES "^.*_parallel$")
if(openPMD_HAVE_MPI)
add_test(NAME MPI.${examplename}
COMMAND ${MPI_TEST_EXE} ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${examplename}
WORKING_DIRECTORY ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}
)
endif()
else()
add_test(NAME Serial.${examplename}
COMMAND ${examplename}
WORKING_DIRECTORY ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}
)
endif()
endforeach()
else()
message(STATUS "Missing samples/git-sample/ directory! "
"Skipping C++ example test!\n\n"
"Note: run\n"
" . ${openPMD_SOURCE_DIR}/.travis/download_samples.sh\n"
"to add example files!")
endif()

# Python Examples
if(openPMD_HAVE_PYTHON)
Expand Down
267 changes: 266 additions & 1 deletion docs/source/dev/backend.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,269 @@
How to Write a Backend
======================

*TBD*
Adding support for additional types of file storage or data transportation is possible by creating a backend.
Backend design has been kept independent of the openPMD-specific logic that maintains all constraints within a file.
This should allow easy introduction of new file formats with only little knowledge about the rest of the system.


File Formats
------------
To get started, you should create a new file format in ``include/openPMD/IO/Format.hpp`` representing the new backend.
Note that this enumeration value will never be seen by users of openPMD-api, but should be kept short and concise to
improve readability.

.. code-block:: cpp
enum class Format
{
JSON
};
In order to use the file format through the API, you need to provide unique and characteristic filename extensions that
are associated with it. This happens in ``src/Series.cpp``:

.. code-block:: cpp
Format
determineFormat(std::string const& filename)
{
if( auxiliary::ends_with(filename, ".json") )
return Format::JSON;
}
.. code-block:: cpp
std::string
cleanFilename(std::string const& filename, Format f)
{
switch( f )
{
case Format::JSON:
return auxiliary::replace_last(filename, ".json", "");
}
}
Unless your file format imposes additional restrictions to the openPMD constraints, this is all you have to do in the
frontend section of the API.

IO Handler
----------
Now that the user can specify that the new backend is to be used, a concrete mechanism for handling IO interactions is
required. We call this an ``IOHandler``. It is not concerned with any logic or constraints enforced by openPMD, but
merely offers a small set of elementary IO operations.

On the very basic level, you will need to derive a class from ``AbstractIOHandler``:

.. code-block:: cpp
/* file: include/openPMD/IO/JSON/JSONIOHandler.hpp */
#include "openPMD/IO/AbstractIOHandler.hpp"
namespace openPMD
{
class JSONIOHandler : public AbstractIOHandler
{
public:
JSONIOHandler(std::string const& path, AccessType);
virtual ~JSONIOHandler();
std::future< void > flush() override;
}
} // openPMD
.. code-block:: cpp
/* file: src/IO/JSON/JSONIOHandler.cpp */
#include "openPMD/IO/JSON/JSONIOHandler.hpp"
namespace openPMD
{
JSONIOHandler::JSONIOHandler(std::string const& path, AccessType at)
: AbstractIOHandler(path, at)
{ }
JSONIOHandler::~JSONIOHandler()
{ }
std::future< void >
JSONIOHandler::flush()
{ return std::future< void >(); }
} // openPMD
Familiarizing your backend with the rest of the API happens in just one place in ``src/IO/AbstractIOHandler.cpp``:

.. code-block:: cpp
#if openPMD_HAVE_MPI
std::shared_ptr< AbstractIOHandler >
AbstractIOHandler::createIOHandler(std::string const& path,
AccessType at,
Format f,
MPI_Comm comm)
{
switch( f )
{
case Format::JSON:
std::cerr << "No MPI-aware JSON backend available. "
"Falling back to the serial backend! "
"Possible failure and degraded performance!" << std::endl;
return std::make_shared< JSONIOHandler >(path, at);
}
}
#endif
std::shared_ptr< AbstractIOHandler >
AbstractIOHandler::createIOHandler(std::string const& path,
AccessType at,
Format f)
{
switch( f )
{
case Format::JSON:
return std::make_shared< JSONIOHandler >(path, at);
}
}
In this state, the backend will do no IO operations and just act as a dummy that ignores all queries.

IO Task Queue
-------------
Operations between the logical representation in this API and physical storage are funneled through a queue ``m_work``
that is contained in the newly created IOHandler. Contained in this queue are ``IOTask`` s that have to be processed in
FIFO order (unless you can prove sequential execution guarantees for out-of-order execution) when
``AbstractIOHandler::flush()`` is called. A **recommended** skeleton is provided in ``AbstractIOHandlerImpl``. Note
that emptying the queue this way is not required and might not fit your IO scheme.

Using the provided skeleton involves
- deriving an IOHandlerImpl for your IOHandler and
- delegating all flush calls to the IOHandlerImpl:

.. code-block:: cpp
/* file: include/openPMD/IO/JSON/JSONIOHandlerImpl.hpp */
#include "openPMD/IO/AbstractIOHandlerImpl.hpp"
namespace openPMD
{
class JSONIOHandlerImpl : public AbstractIOHandlerImpl
{
public:
JSONIOHandlerImpl(AbstractIOHandler*);
virtual ~JSONIOHandlerImpl();
virtual void createFile(Writable*, Parameter< Operation::CREATE_FILE > const&) override;
virtual void createPath(Writable*, Parameter< Operation::CREATE_PATH > const&) override;
virtual void createDataset(Writable*, Parameter< Operation::CREATE_DATASET > const&) override;
virtual void extendDataset(Writable*, Parameter< Operation::EXTEND_DATASET > const&) override;
virtual void openFile(Writable*, Parameter< Operation::OPEN_FILE > const&) override;
virtual void openPath(Writable*, Parameter< Operation::OPEN_PATH > const&) override;
virtual void openDataset(Writable*, Parameter< Operation::OPEN_DATASET > &) override;
virtual void deleteFile(Writable*, Parameter< Operation::DELETE_FILE > const&) override;
virtual void deletePath(Writable*, Parameter< Operation::DELETE_PATH > const&) override;
virtual void deleteDataset(Writable*, Parameter< Operation::DELETE_DATASET > const&) override;
virtual void deleteAttribute(Writable*, Parameter< Operation::DELETE_ATT > const&) override;
virtual void writeDataset(Writable*, Parameter< Operation::WRITE_DATASET > const&) override;
virtual void writeAttribute(Writable*, Parameter< Operation::WRITE_ATT > const&) override;
virtual void readDataset(Writable*, Parameter< Operation::READ_DATASET > &) override;
virtual void readAttribute(Writable*, Parameter< Operation::READ_ATT > &) override;
virtual void listPaths(Writable*, Parameter< Operation::LIST_PATHS > &) override;
virtual void listDatasets(Writable*, Parameter< Operation::LIST_DATASETS > &) override;
virtual void listAttributes(Writable*, Parameter< Operation::LIST_ATTS > &) override;
}
} // openPMD
.. code-block:: cpp
/* file: include/openPMD/IO/JSON/JSONIOHandler.hpp */
#include "openPMD/IO/AbstractIOHandler.hpp"
#include "openPMD/IO/JSON/JSONIOHandlerImpl.hpp"
namespace openPMD
{
class JSONIOHandler : public AbstractIOHandler
{
public:
/* ... */
private:
JSONIOHandlerImpl m_impl;
}
} // openPMD
.. code-block:: cpp
/* file: src/IO/JSON/JSONIOHandler.cpp */
#include "openPMD/IO/JSON/JSONIOHandler.hpp"
namespace openPMD
{
/*...*/
std::future< void >
JSONIOHandler::flush()
{
return m_impl->flush();
}
} // openPMD
Each IOTask contains a pointer to a ``Writable`` that corresponds to one object in the openPMD hierarchy. This object
may be a group or a dataset. When processing certain types of IOTasks in the queue, you will have to assign unique
FilePositions to these objects to identify the logical object in your physical storage. For this, you need to derive
a concrete FilePosition for your backend from ``AbstractFilePosition``. There is no requirement on how to identify your
objects, but ids from your IO library and positional strings are good candidates.

.. code-block:: cpp
/* file: include/openPMD/IO/JSON/JSONFilePosition.hpp */
#include "openPMD/IO/AbstractFilePosition.hpp"
namespace openPMD
{
struct JSONFilePosition : public AbstractFilePosition
{
JSONFilePosition(uint64_t id)
: id{id}
{ }
uint64_t id;
};
} // openPMD
From this point, all that is left to do is implement the elementary IO operations provided in the IOHandlerImpl. The
``Parameter`` structs contain both input parameters (from storage to API) and output parameters (from API to storage).
The easy way to distinguish between the two parameter sets is their C++ type: Input parameters are
``std::shared_ptr`` s that allow you to pass the requested data to their destination. Output parameters are all objects
that are *not* ``std::shared_ptr`` s. The contract of each function call is outlined in
``include/openPMD/IO/AbstractIOHandlerImpl``.

.. code-block:: cpp
/* file: src/IO/JSON/JSONIOHandlerImpl.cpp */
#include "openPMD/IO/JSONIOHandlerImpl.hpp"
namespace openPMD
{
void
JSONIOHandlerImpl::createFile(Writable* writable,
Parameter< Operation::CREATE_FILE > const& parameters)
{
if( !writable->written )
{
path dir(m_handler->directory);
if( !exists(dir) )
create_directories(dir);
std::string name = m_handler->directory + parameters.name;
if( !auxiliary::ends_with(name, ".json") )
name += ".json";
uint64_t id = /*...*/
ASSERT(id >= 0, "Internal error: Failed to create JSON file");
writable->written = true;
writable->abstractFilePosition = std::make_shared< JSONFilePosition >(id);
}
}
/*...*/
} // openPMD
Note that you might have to keep track of open file handles if they have to be closed explicitly during destruction of
the IOHandlerImpl (prominent in C-style frameworks).
2 changes: 1 addition & 1 deletion include/openPMD/IO/AbstractIOHandler.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ class AbstractIOHandler
{
public:
#if openPMD_HAVE_MPI
/** Construct an appropriate specific IOHandler for the desired IO mode.
/** Construct an appropriate specific IOHandler for the desired IO mode that may be MPI-aware.
*
* @param path Path to root folder for all operations associated with the desired handler.
* @param accessType AccessType describing desired operations and permissions of the desired handler.
Expand Down
Loading

0 comments on commit aea4659

Please sign in to comment.