From f4d6a632635c9429dd93cb56174d28fc654c75bc Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB?=
<41448637+mkgrgis@users.noreply.github.com>
Date: Wed, 4 Oct 2023 09:56:16 +0300
Subject: [PATCH 1/3] Refactor README.md for PGXN
Add separate `INSTALL.md`.
Other changes in `README.md` for unifying with https://github.com/pgspider FDW documentation template and by https://github.com/ibarwick/firebird_fdw/blob/master/README.md example of advanced FDW documentation.
---
INSTALL.md | 205 ++++++++++++++++
README.md | 669 ++++++++++++++++++++++++++++-------------------------
2 files changed, 555 insertions(+), 319 deletions(-)
create mode 100644 INSTALL.md
diff --git a/INSTALL.md b/INSTALL.md
new file mode 100644
index 0000000..624b800
--- /dev/null
+++ b/INSTALL.md
@@ -0,0 +1,205 @@
+Notes about installation Mongo Foreign Data Wrapper
+===================================================
+
+To compile the [MongoDB][1] foreign data wrapper for [PostgreSQL](https://www.postgresql.org/), `mongo-c` and `json-c`
+libraries are needed. To build and install `mongo-c` and `json-c` libraries, there
+are two ways. You can either use script `autogen.sh` or you can manually
+perform all required steps listed.
+
+### Notes about new MongoDB C Driver support
+This enhancement is to add a new [MongoDB][1]' C driver. The current
+implementation is based on the legacy driver of MongoDB. But
+[MongoDB][1] is provided completely new library for driver called
+MongoDB's meta driver. Added support for the same. Now compile time
+option is available to use legacy and meta driver.
+
+In order to use MongoDB driver 1.17.0+, take the following steps:
+
+ * clone `libmongoc` version 1.17.0+
+ (https://github.com/mongodb/mongo-c-driver) and follow the install
+ directions given there. `libbson` is now maintained in a subdirectory
+ of the `libmongoc`.
+ (https://github.com/mongodb/mongo-c-driver/tree/master/src/libbson).
+ * ensure pkg-config / pkgconf is installed on your system.
+ * run `make -f Makefile.meta && make -f Makefile.meta install`
+ * if you get an error when trying to `CREATE EXTENSION `mongo_fdw`;`,
+ then try running `ldconfig`
+
+## Installation using script
+Number of manual steps needs to be performed to compile and install required
+mongo-c and json-c libraries. If you want to avoid the manual steps, there is a
+shell script available which will download and install the appropriate drivers
+and libraries for you.
+
+Here is how it works:
+
+To install mongo-c and json-c libraries at custom locations, you need to
+export environment variables `MONGOC_INSTALL_DIR` and `JSONC_INSTALL_DIR`
+respectively. If these variables are not set then these libraries will be
+installed in the default location. Please note that you need to have the
+required permissions on the directory where you want to install the libraries.
+
+Build with [MongoDB][1]'s legacy branch driver (Deprecated in mongo_fdw 5.4.0)
+ * autogen.sh --with-legacy
+
+Build [MongoDB][1]'s master branch driver
+ * autogen.sh --with-master
+
+The script autogen.sh will do all the necessary steps to build with legacy and
+meta driver accordingly.
+
+## Steps for manual installation
+### mongo-c
+#### meta driver
+1. Download and extract source code of mongoc driver for version `1.17.3`
+
+ ```sh
+ wget https://github.com/mongodb/mongo-c-driver/releases/download/1.17.3/mongo-c-driver-1.17.3.tar.gz
+ tar xzf mongo-c-driver-1.17.3.tar.gz
+ rm -rf mongo-c-driver
+ mv mongo-c-driver-1.17.3 mongo-c-driver
+ cd mongo-c-driver
+ ```
+
+2. Configure mongoc driver
+
+ ```sh
+ cmake -DENABLE_AUTOMATIC_INIT_AND_CLEANUP=OFF .
+ ```
+
+ To install at custom location:
+
+ ```sh
+ cmake -DCMAKE_INSTALL_PREFIX=YOUR_INSTALLATION_DIRECTORY -DENABLE_AUTOMATIC_INIT_AND_CLEANUP=OFF .
+ ```
+
+3. Compile and install
+
+ ```sh
+ cmake --build .
+ cmake --build . --target install
+ ```
+
+For more details on installation of mongo-c driver, you can refer [here][3].
+
+#### Legacy driver
+
+Deprecation Notice:
+The legacy driver support has been deprecated in mongo_fdw 5.4.0 and is
+expected to be removed entirely in a future release.
+
+* Checkout, extract legacy branch
+
+ ```sh
+ wget https://github.com/mongodb/mongo-c-driver/archive/v0.8.tar.gz
+ tar -zxf v0.8.tar.gz
+ rm -rf mongo-c-driver
+ mv mongo-c-driver-0.8 mongo-c-driver
+ ```
+
+### json-c
+
+1. Download and extract source code
+
+ ```sh
+ wget https://github.com/json-c/json-c/archive/json-c-0.15-20200726.tar.gz
+ tar -xzf json-c-0.15-20200726.tar.gz
+ rm -rf json-c
+ mv json-c-json-c-0.15-20200726/ json-c
+ cd json-c
+ ```
+
+2. Configure
+
+ ```sh
+ cmake .
+ ```
+ To install at custom location:
+
+ ```sh
+ cmake -DCMAKE_INSTALL_PREFIX=YOUR_INSTALLATION_DIRECTORY .
+ ```
+
+3. Compile and install
+
+ ```sh
+ make
+ make install
+ ```
+
+For more details on installation of json-c library, you can refer [here][4].
+
+### How to compile against mongo-c Meta or Legacy driver?
+To compile against legacy driver, 'Makefile.legacy' must be used and
+'Makefile.meta' must be used to compile against the meta driver. For example,
+this can be achieved by copying required Makefile as shown below:
+For meta,
+
+ cp Makefile.meta Makefile
+
+For legacy (Deprecated in mongo_fdw 5.4.0),
+
+ cp Makefile.legacy Makefile
+
+The default compilation is with Meta driver.
+
+## Mongo_fdw configuration, compilation and installation
+The `PKG_CONFIG_PATH` environment variable must be set to mongo-c-driver source
+directory for successful compilation as shown below,
+
+```sh
+export PKG_CONFIG_PATH=$YOUR_MONGO_FDW_SOURCE_DIR/mongo-c-driver/src/libmongoc/src:$YOUR_MONGO_FDW_SOURCE_DIR/mongo-c-driver/src/libbson/src
+```
+
+The `LD_LIBRARY_PATH` environment variable must include the path to the mongo-c
+installation directory containing the libmongoc-1.0.so and libbson-1.0.so
+files. For example, assuming the installation directory is /home/mongo-c and
+the libraries were created under it in lib64 sub-directory, then we can define
+the `LD_LIBRARY_PATH` as:
+
+```sh
+export LD_LIBRARY_PATH=/home/mongo-c/lib64:$LD_LIBRARY_PATH
+```
+
+Note: This `LD_LIBRARY_PATH` environment variable setting must be in effect
+when the `pg_ctl` utility is executed to start or restart PostgreSQL or
+EDB Postgres Advanced Server.
+
+
+1. To build on POSIX-compliant systems you need to ensure the
+ `pg_config` executable is in your path when you run `make`. This
+ executable is typically in your PostgreSQL installation's `bin`
+ directory. For example:
+
+ ```sh
+ export PATH=/usr/local/pgsql/bin/:$PATH
+ ```
+
+2. Compile the code using make.
+
+ ```sh
+ make USE_PGXS=1
+ ```
+
+3. Finally install the foreign data wrapper.
+
+ ```sh
+ make USE_PGXS=1 install
+ ```
+
+4. Running regression test.
+
+ ```sh
+ make USE_PGXS=1 installcheck
+ ```
+ However, make sure to set the `MONGO_HOST`, `MONGO_PORT`, `MONGO_USER_NAME`,
+ and `MONGO_PWD` environment variables correctly. The default settings can be
+ found in the `mongodb_init.sh` script.
+
+
+If you run into any issues, please [let us know][2].
+
+[1]: http://www.mongodb.com
+[2]: https://github.com/enterprisedb/mongo_fdw/issues/new
+[3]: http://mongoc.org/libmongoc/1.17.3/installing.html#configuring-the-build
+[4]: https://github.com/json-c/json-c/tree/json-c-0.15-20200726#build-instructions--
diff --git a/README.md b/README.md
index 8b31b8d..816c823 100644
--- a/README.md
+++ b/README.md
@@ -1,348 +1,296 @@
-# MongoDB Foreign Data Wrapper for PostgreSQL
+MongoDB Foreign Data Wrapper for PostgreSQL
+============================================
This PostgreSQL extension implements a Foreign Data Wrapper (FDW) for
[MongoDB][1].
Please note that this version of mongo_fdw works with PostgreSQL and EDB
-Postgres Advanced Server 11, 12, 13, 14, 15, and 16.
+Postgres Advanced Server 11, 12, 13, 14, 15 and 16.
+
+
+ 
+
+Contents
+--------
+
+1. [Features](#features)
+2. [Supported platforms](#supported-platforms)
+3. [Installation](#installation)
+4. [Usage](#usage)
+5. [Functions](#functions)
+6. [Identifier case handling](#identifier-case-handling)
+7. [Generated columns](#generated-columns)
+8. [Character set handling](#character-set-handling)
+9. [Examples](#examples)
+10. [Limitations](#limitations)
+11. [Contributing and testing](#contributing-and-testing)
+12. [Useful links](#useful-links)
+
+Features
+--------
+
+The following enhancements are added to the latest version of `mongo_fdw`:
+
+#### Write-able FDW
+The previous version was only read-only, the latest version provides the
+write capability. The user can now issue an insert, update, and delete
+statements for the foreign tables using the `mongo_fdw`.
+
+#### Connection Pooling
+The latest version comes with a connection pooler that utilizes the
+same MongoDB database connection for all the queries in the same session.
+The previous version would open a new [MongoDB][1] connection for every
+query. This is a performance enhancement.
+
+#### JOIN push-down
+`mongo_fdw` now also supports join push-down. The joins between two
+foreign tables from the same remote MySQL server are pushed to a remote
+server, instead of fetching all the rows for both the tables and
+performing a join locally, thereby may enhance the performance. Currently,
+joins involving only relational and arithmetic operators in join-clauses
+are pushed down to avoid any potential join failure. Also, only the
+INNER and LEFT/RIGHT OUTER joins are supported, and not the FULL OUTER,
+SEMI, and ANTI join. Moreover, only joins between two tables are pushed
+down and not when either inner or outer relation is the join itself.
+
+#### AGGREGATE push-down
+`mongo_fdw` now also supports aggregate push-down. Push aggregates to the
+remote MongoDB server instead of fetching all of the rows and aggregating
+them locally. This gives a very good performance boost for the cases
+where aggregates can be pushed down. The push-down is currently limited
+to aggregate functions min, max, sum, avg, and count, to avoid pushing
+down the functions that are not present on the MongoDB server. The
+aggregate filters, orders, variadic and distinct are not pushed down.
+
+#### ORDER BY push-down
+`mongo_fdw` now also supports order by push-down. If possible, push order
+by clause to the remote server so that we get the ordered result set from
+the foreign server itself. It might help us to have an efficient merge
+join. NULLs behavior is opposite on the MongoDB server. Thus to get an
+equivalent result, we can only push-down ORDER BY with either
+ASC NULLS FIRST or DESC NULLS LAST. Moreover, as MongoDB sorts only on
+fields, only column names in ORDER BY expressions are pushed down.
+
+#### LIMIT OFFSET push-down
+`mongo_fdw` now also supports limit offset push-down. Wherever possible,
+perform LIMIT and OFFSET operations on the remote server. This reduces
+network traffic between local PostgreSQL and remote MongoDB servers.
+
+#### GUC variables:
+
+ * `mongo_fdw.enable_order_by_pushdown`: If `true`, pushes the order by
+ operation to the foreign server, instead of fetching rows from the
+ foreign server and performing the sort locally. Default is `true`.
+ * `mongo_fdw.enable_join_pushdown`: If `true`, pushes the join between two
+ foreign tables from the same foreign server, instead of fetching all the
+ rows for both the tables and performing a join locally. Default is `true`.
+ * `mongo_fdw.enable_aggregate_pushdown`: If `true`, pushes aggregate
+ operations to the foreign server, instead of fetching rows from the
+ foreign server and performing the operations locally. Default is `true`.
+
+Supported platforms
+-------------------
+
+`mongo_fdw` was developed on Linux, and should run on any
+reasonably POSIX-compliant system.
Installation
------------
-To compile the [MongoDB][1] foreign data wrapper, mongo-c and json-c
-libraries are needed. To build and install mongo-c and json-c libraries, there
-are two ways. You can either use script `autogen.sh` or you can manually
-perform all required steps listed.
-## Installation using script
-Number of manual steps needs to be performed to compile and install required
-mongo-c and json-c libraries. If you want to avoid the manual steps, there is a
-shell script available which will download and install the appropriate drivers
-and libraries for you.
+About script or manual installation, `mongo-c`, legacy and `meta` driver please read the following [instructions in INSTALL.md](INSTALL.md).
-Here is how it works:
+If you run into any issues, please [let us know][2].
-To install mongo-c and json-c libraries at custom locations, you need to
-export environment variables `MONGOC_INSTALL_DIR` and `JSONC_INSTALL_DIR`
-respectively. If these variables are not set then these libraries will be
-installed in the default location. Please note that you need to have the
-required permissions on the directory where you want to install the libraries.
+Usage
+-----
-Build with [MongoDB][1]'s legacy branch driver (Deprecated in mongo_fdw 5.4.0)
- * autogen.sh --with-legacy
+## CREATE SERVER options
-Build [MongoDB][1]'s master branch driver
- * autogen.sh --with-master
+`mongo_fdw` accepts the following options via the `CREATE SERVER` command:
-The script autogen.sh will do all the necessary steps to build with legacy and
-meta driver accordingly.
+- **address** as *string*, optional, default `127.0.0.1`
-## Steps for manual installation
-### mongo-c
-#### meta driver
-1. Download and extract source code of mongoc driver for version `1.17.3`
+ Address or hostname of the MongoDB server.
- ```sh
- wget https://github.com/mongodb/mongo-c-driver/releases/download/1.17.3/mongo-c-driver-1.17.3.tar.gz
- tar xzf mongo-c-driver-1.17.3.tar.gz
- rm -rf mongo-c-driver
- mv mongo-c-driver-1.17.3 mongo-c-driver
- cd mongo-c-driver
- ```
+- **port** as *integer*, optional, default `27017`.
-2. Configure mongoc driver
+ Port number of the MongoDB server.
- ```sh
- cmake -DENABLE_AUTOMATIC_INIT_AND_CLEANUP=OFF .
- ```
+- **use_remote_estimate** as *boolean*, optional, default `false`
- To install at custom location:
+ Controls whether `mongo_fdw` uses exact rows from
+ remote collection to obtain cost estimates.
- ```sh
- cmake -DCMAKE_INSTALL_PREFIX=YOUR_INSTALLATION_DIRECTORY -DENABLE_AUTOMATIC_INIT_AND_CLEANUP=OFF .
- ```
+- **enable_order_by_pushdown** as *boolean*, optional, default `true`
-3. Compile and install
+ If `true`, pushes the ORDER BY clause to theforeign server instead of
+ performing a sort locally. This option can also be set for an individual
+ table, and if any of the tables involved in the query has set it to
+ false then the ORDER BY will not be pushed down. The table-level value
+ of the option takes precedence over the server-level option value.
- ```sh
- cmake --build .
- cmake --build . --target install
- ```
+The following options are _only supported with meta driver_:
-For more details on installation of mongo-c driver, you can refer [here][5].
+- **authentication_database** as *string*, optional
-#### Legacy driver
+ Database against which user will be
+ authenticated against. Only valid with password based authentication.
-Deprecation Notice:
-The legacy driver support has been deprecated in mongo_fdw 5.4.0 and is
-expected to be removed entirely in a future release.
+- **replica_set** as *string*, optional
-* Checkout, extract legacy branch
+ Replica set the server is member of. If set,
+ driver will auto-connect to correct primary in the replica set when
+ writing.
- ```sh
- wget https://github.com/mongodb/mongo-c-driver/archive/v0.8.tar.gz
- tar -zxf v0.8.tar.gz
- rm -rf mongo-c-driver
- mv mongo-c-driver-0.8 mongo-c-driver
- ```
+- **read_preference** as *string*, optional, default `primary`
-### json-c
-1. Download and extract source code
+ `primary`, `secondary`, `primaryPreferred`,
+ `secondaryPreferred`, or `nearest`.
- ```sh
- wget https://github.com/json-c/json-c/archive/json-c-0.15-20200726.tar.gz
- tar -xzf json-c-0.15-20200726.tar.gz
- rm -rf json-c
- mv json-c-json-c-0.15-20200726/ json-c
- cd json-c
- ```
+- **ssl** as *boolean*, optional, default `false`
-2. Configure
+ Enable ssl. See http://mongoc.org/libmongoc/current/mongoc_ssl_opt_t.html to
+ understand the options.
- ```sh
- cmake .
- ```
- To install at custom location:
+- **pem_file** as *string*, optional
- ```sh
- cmake -DCMAKE_INSTALL_PREFIX=YOUR_INSTALLATION_DIRECTORY .
- ```
+ The .pem file that contains both the TLS/SSL certificate and
+ key.
-3. Compile and install
+- **pem_pwd** as *string*, optional
- ```sh
- make
- make install
- ```
+ The password to decrypt the certificate key file(i.e. pem_file)
-For more details on installation of json-c library, you can refer [here][6].
+- **ca_file** as *string*, optional
-### How to compile against mongo-c Meta or Legacy driver?
-To compile against legacy driver, 'Makefile.legacy' must be used and
-'Makefile.meta' must be used to compile against the meta driver. For example,
-this can be achieved by copying required Makefile as shown below:
-For meta,
+ The .pem file that contains the root certificate chain from the
+ Certificate Authority.
- cp Makefile.meta Makefile
+- **ca_dir** as *string*, optional
-For legacy (Deprecated in mongo_fdw 5.4.0),
+ The absolute path to the `ca_file`.
- cp Makefile.legacy Makefile
+- **crl_file** as *string*, optional
-The default compilation is with Meta driver.
+ The .pem file that contains the Certificate Revocation List.
-## Mongo_fdw configuration, compilation and installation
-The `PKG_CONFIG_PATH` environment variable must be set to mongo-c-driver source
-directory for successful compilation as shown below,
+- **weak_cert_validation** as *boolean*, optional, default `false`
-```sh
-export PKG_CONFIG_PATH=$YOUR_MONGO_FDW_SOURCE_DIR/mongo-c-driver/src/libmongoc/src:$YOUR_MONGO_FDW_SOURCE_DIR/mongo-c-driver/src/libbson/src
-```
+ Enable the validation checks for TLS/SSL certificates and allows the use of invalid
+ certificates to connect if set to `true`.
-The `LD_LIBRARY_PATH` environment variable must include the path to the mongo-c
-installation directory containing the libmongoc-1.0.so and libbson-1.0.so
-files. For example, assuming the installation directory is /home/mongo-c and
-the libraries were created under it in lib64 sub-directory, then we can define
-the `LD_LIBRARY_PATH` as:
+- **enable_join_pushdown** as *boolean*, optional, default `true`
-```sh
-export LD_LIBRARY_PATH=/home/mongo-c/lib64:$LD_LIBRARY_PATH
-```
+ If `true`, pushes the join between two foreign
+ tables from the same foreign server, instead of fetching all the rows
+ for both the tables and performing a join locally. This option can also
+ be set for an individual table, and if any of the tables involved in the
+ join has set it to false then the join will not be pushed down. The
+ table-level value of the option takes precedence over the server-level
+ option value.
-Note: This `LD_LIBRARY_PATH` environment variable setting must be in effect
-when the `pg_ctl` utility is executed to start or restart PostgreSQL or
-EDB Postgres Advanced Server.
+- **enable_aggregate_pushdown** as *boolean*, optional, default `true`
+ If `true`, push aggregates to the remote
+ MongoDB server instead of fetching all of the rows and aggregating them
+ locally. This option can also be set for an individual table. The
+ table-level value of the option takes precedence over the server-level
+ option value.
-1. To build on POSIX-compliant systems you need to ensure the
- `pg_config` executable is in your path when you run `make`. This
- executable is typically in your PostgreSQL installation's `bin`
- directory. For example:
+## CREATE USER MAPPING options
- ```sh
- export PATH=/usr/local/pgsql/bin/:$PATH
- ```
+`mongo_fdw` accepts the following options via the `CREATE USER MAPPING`
+command:
-2. Compile the code using make.
+- **username** as *string*, optional
- ```sh
- make USE_PGXS=1
- ```
+ Username to use when connecting to MongoDB.
-3. Finally install the foreign data wrapper.
+- **password** as *string*, optional
- ```sh
- make USE_PGXS=1 install
- ```
+ Password to authenticate to the MongoDB server.
-4. Running regression test.
+## CREATE FOREIGN TABLE options
- Run `mongodb_init.sh` file to load required collections.
+`mongo_fdw` accepts the following table-level options via the
+`CREATE FOREIGN TABLE` command:
- ```sh
- source mongodb_init.sh
- ```
- Finally, run regression.
+- **database** as *string*, optional, default `test`
- ```sh
- make USE_PGXS=1 installcheck
- ```
- However, make sure to set the `MONGO_HOST`, `MONGO_PORT`, `MONGO_USER_NAME`,
- and `MONGO_PWD` environment variables correctly. The default settings can
- be found in the `mongodb_init.sh` script.
+ Name of the MongoDB database to query.
+- **collection** as *string*, optional, default name of foreign table
-If you run into any issues, please [let us know][2].
+ Name of the MongoDB collection to query.
-Enhancements
------------
-The following enhancements are added to the latest version of mongo_fdw:
+- **enable_join_pushdown** as *boolean*, optional, default `true`
-### Write-able FDW
-The previous version was only read-only, the latest version provides the
-write capability. The user can now issue an insert, update, and delete
-statements for the foreign tables using the mongo_fdw.
+ Similar to the server-level option, but can be
+ configured at table level as well.
-### Connection Pooling
-The latest version comes with a connection pooler that utilizes the
-same MongoDB database connection for all the queries in the same session.
-The previous version would open a new [MongoDB][1] connection for every
-query. This is a performance enhancement.
+- **enable_aggregate_pushdown** as *boolean*, optional, default corresponding servel-lelel value
-### JOIN push-down
-mongo_fdw now also supports join push-down. The joins between two
-foreign tables from the same remote MySQL server are pushed to a remote
-server, instead of fetching all the rows for both the tables and
-performing a join locally, thereby may enhance the performance. Currently,
-joins involving only relational and arithmetic operators in join-clauses
-are pushed down to avoid any potential join failure. Also, only the
-INNER and LEFT/RIGHT OUTER joins are supported, and not the FULL OUTER,
-SEMI, and ANTI join. Moreover, only joins between two tables are pushed
-down and not when either inner or outer relation is the join itself.
+ Similar to the server-level option, but can be configured at table level as well.
-### AGGREGATE push-down
-mongo_fdw now also supports aggregate push-down. Push aggregates to the
-remote MongoDB server instead of fetching all of the rows and aggregating
-them locally. This gives a very good performance boost for the cases
-where aggregates can be pushed down. The push-down is currently limited
-to aggregate functions min, max, sum, avg, and count, to avoid pushing
-down the functions that are not present on the MongoDB server. The
-aggregate filters, orders, variadic and distinct are not pushed down.
+- **enable_order_by_pushdown** as *boolean*, optional, default corresponding servel-lelel value
-### ORDER BY push-down
-mongo_fdw now also supports order by push-down. If possible, push order
-by clause to the remote server so that we get the ordered result set from
-the foreign server itself. It might help us to have an efficient merge
-join. NULLs behavior is opposite on the MongoDB server. Thus to get an
-equivalent result, we can only push-down ORDER BY with either
-ASC NULLS FIRST or DESC NULLS LAST. Moreover, as MongoDB sorts only on
-fields, only column names in ORDER BY expressions are pushed down.
+ Similar to the server-level option, but can be configured at table level as well.
-### LIMIT OFFSET push-down
-mongo_fdw now also supports limit offset push-down. Wherever possible,
-perform LIMIT and OFFSET operations on the remote server. This reduces
-network traffic between local PostgreSQL and remote MongoDB servers.
+No column-level options are available.
-### New MongoDB C Driver Support
-This enhancement is to add a new [MongoDB][1]' C driver. The current
-implementation is based on the legacy driver of MongoDB. But
-[MongoDB][1] is provided completely new library for driver called
-MongoDB's meta driver. Added support for the same. Now compile time
-option is available to use legacy and meta driver.
-
-In order to use MongoDB driver 1.17.0+, take the following steps:
-
- * clone `libmongoc` version 1.17.0+
- (https://github.com/mongodb/mongo-c-driver) and follow the install
- directions given there. `libbson` is now maintained in a subdirectory
- of the `libmongoc`.
- (https://github.com/mongodb/mongo-c-driver/tree/master/src/libbson).
- * ensure pkg-config / pkgconf is installed on your system.
- * run `make -f Makefile.meta && make -f Makefile.meta install`
- * if you get an error when trying to `CREATE EXTENSION mongo_fdw;`,
- then try running `ldconfig`
+## IMPORT FOREIGN SCHEMA options
-Usage
------
-The following parameters can be set on a MongoDB foreign server object:
+`mongo_fdw` don't supports [IMPORT FOREIGN SCHEMA](https://www.postgresql.org/docs/current/sql-importforeignschema.html)
+because MongoDB is schemaless.
- * `address`: Address or hostname of the MongoDB server. Defaults to
- `127.0.0.1`
- * `port`: Port number of the MongoDB server. Defaults to `27017`.
- * `use_remote_estimate`: Controls whether mongo_fdw uses exact rows from
- remote collection to obtain cost estimates. Default is `false`.
+## TRUNCATE support
-The following options are only supported with meta driver:
+`mongo_fdw` don't implements the foreign data wrapper `TRUNCATE` API, available
+from PostgreSQL 14, because MongoDB is schemaless.
- * `authentication_database`: Database against which user will be
- authenticated against. Only valid with password based authentication.
- * `replica_set`: Replica set the server is member of. If set,
- driver will auto-connect to correct primary in the replica set when
- writing.
- * `read_preference`: primary [default], secondary, primaryPreferred,
- secondaryPreferred, or nearest.
- * `ssl`: false [default], true to enable ssl. See
- http://mongoc.org/libmongoc/current/mongoc_ssl_opt_t.html to
- understand the options.
- * `pem_file`: The .pem file that contains both the TLS/SSL certificate and
- key.
- * `pem_pwd`: The password to decrypt the certificate key file(i.e. pem_file)
- * `ca_file`: The .pem file that contains the root certificate chain from the
- Certificate Authority.
- * `ca_dir`: The absolute path to the `ca_file`.
- * `crl_file`: The .pem file that contains the Certificate Revocation List.
- * `weak_cert_validation`: false [default], This is to enable or disable the
- validation checks for TLS/SSL certificates and allows the use of invalid
- certificates to connect if set to `true`.
- * `enable_join_pushdown`: If `true`, pushes the join between two foreign
- tables from the same foreign server, instead of fetching all the rows
- for both the tables and performing a join locally. This option can also
- be set for an individual table, and if any of the tables involved in the
- join has set it to false then the join will not be pushed down. The
- table-level value of the option takes precedence over the server-level
- option value. Default is `true`.
- * `enable_aggregate_pushdown`: If `true`, push aggregates to the remote
- MongoDB server instead of fetching all of the rows and aggregating them
- locally. This option can also be set for an individual table. The
- table-level value of the option takes precedence over the server-level
- option value. Default is `true`.
- * `enable_order_by_pushdown`: If `true`, pushes the ORDER BY clause to the
- foreign server instead of performing a sort locally. This option can also
- be set for an individual table, and if any of the tables involved in the
- query has set it to false then the ORDER BY will not be pushed down. The
- table-level value of the option takes precedence over the server-level
- option value. Default is `true`.
+Functions
+---------
-The following parameters can be set on a MongoDB foreign table object:
+As well as the standard `mongo_fdw_handler()` and `mongo_fdw_validator()`
+functions, `mongo_fdw` provides the following user-callable utility functions:
- * `database`: Name of the MongoDB database to query. Defaults to
- `test`.
- * `collection`: Name of the MongoDB collection to query. Defaults to
- the foreign table name used in the relevant `CREATE` command.
- * `enable_join_pushdown`: Similar to the server-level option, but can be
- configured at table level as well. Default is `true`.
- * `enable_aggregate_pushdown`: Similar to the server-level option, but
- can be configured at table level as well. Default is `true`.
- * `enable_order_by_pushdown`: Similar to the server-level option, but can
- be configured at table level as well. Default is `true`.
+**Yet not described!**.
-The following parameters can be supplied while creating user mapping:
+Identifier case handling
+------------------------
- * `username`: Username to use when connecting to MongoDB.
- * `password`: Password to authenticate to the MongoDB server.
+PostgreSQL folds identifiers to lower case by default, MongoDB use JSON notation of identifiers.
-GUC variables:
+All transformation rules and problems **yet not described**.
- * `mongo_fdw.enable_join_pushdown`: If `true`, pushes the join between two
- foreign tables from the same foreign server, instead of fetching all the
- rows for both the tables and performing a join locally. Default is `true`.
+Generated columns
+-----------------
- * `mongo_fdw.enable_aggregate_pushdown`: If `true`, pushes aggregate
- operations to the foreign server, instead of fetching rows from the
- foreign server and performing the operations locally. Default is `true`.
+`mongo_fdw` does't provides support for PostgreSQL's generated
+columns (PostgreSQL 12+).
- * `mongo_fdw.enable_order_by_pushdown`: If `true`, pushes the order by
- operation to the foreign server, instead of fetching rows from the
- foreign server and performing the sort locally. Default is `true`.
+**Behaviour with generated columns yet not tested and not described**.
+
+Note that while `mongo_fdw` will `INSERT` or `UPDATE` the generated column value
+in MongoDB, there is nothing to stop the value being modified within MongoDB,
+and hence no guarantee that in subsequent `SELECT` operations the column will
+still contain the expected generated value. This limitation also applies to
+`postgres_fdw`.
+
+For more details on generated columns see:
+
+- [Generated Columns](https://www.postgresql.org/docs/current/ddl-generated-columns.html)
+- [CREATE FOREIGN TABLE](https://www.postgresql.org/docs/current/sql-createforeigntable.html)
+
+Character set handling
+----------------------
+
+`BSON` in MongoDB can only be encoded in `UTF-8`. Also `UTF-8` is recommended and
+de-facto most popular PostgreSQL server encoding.
+
+Encodings mapping between PostgreSQL and MongoDB **yet not described**.
+
+Examples
+--------
As an example, the following commands demonstrate loading the
`mongo_fdw` wrapper, creating a server, and then creating a foreign
@@ -354,45 +302,85 @@ isn't provided, the wrapper uses the default value mentioned above.
them when estimating costs for the query execution plan. To see selected
execution plans for a query, just run `EXPLAIN`.
-Examples
---------
+### Install the extension:
-Examples with [MongoDB][1]'s equivalent statements.
+Once for a database you need, as PostgreSQL superuser.
```sql
--- load extension first time after install
-CREATE EXTENSION mongo_fdw;
+ CREATE EXTENSION mongo_fdw;
+```
--- create server object
-CREATE SERVER mongo_server
+### Create a foreign server with appropriate configuration:
+
+Once for a foreign datasource you need, as PostgreSQL superuser.
+
+```sql
+ CREATE SERVER "MongoDB server"
FOREIGN DATA WRAPPER mongo_fdw
- OPTIONS (address '127.0.0.1', port '27017');
-
--- create user mapping
-CREATE USER MAPPING FOR postgres
- SERVER mongo_server
- OPTIONS (username 'mongo_user', password 'mongo_pass');
-
--- create foreign table
-CREATE FOREIGN TABLE warehouse
- (
- _id name,
- warehouse_id int,
- warehouse_name text,
- warehouse_created timestamptz
+ OPTIONS (
+ address '127.0.0.1',
+ port '27017'
+ );
+```
+
+### Grant usage on foreign server to normal user in PostgreSQL:
+
+Once for a normal user (non-superuser) in PostgreSQL, as PostgreSQL superuser. It is a good idea to use a superuser only where really necessary, so let's allow a normal user to use the foreign server (this is not required for the example to work, but it's secirity recomedation).
+
+```sql
+ GRANT USAGE ON FOREIGN SERVER "MongoDB server" TO pguser;
+```
+Where `pguser` is a sample user for works with foreign server (and foreign tables).
+
+### User mapping
+
+Create an appropriate user mapping:
+```sql
+ CREATE USER MAPPING
+ FOR pguser
+ SERVER "MongoDB server"
+ OPTIONS (
+ username 'mongo_user',
+ password 'mongo_pass'
+ );
+```
+Where `pguser` is a sample user for works with foreign server (and foreign tables).
+
+### Create foreign table
+All `CREATE FOREIGN TABLE` SQL commands can be executed as a normal PostgreSQL user if there were correct `GRANT USAGE ON FOREIGN SERVER`. No need PostgreSQL supersuer for secirity reasons but also works with PostgreSQL supersuer.
+
+Create a foreign table referencing the MongoDB collection:
+
+```sql
+ -- Note: first column of the table must be "_id" of type "name".
+ CREATE FOREIGN TABLE warehouse (
+ _id name,
+ warehouse_id int,
+ warehouse_name text,
+ warehouse_created timestamptz
)
- SERVER mongo_server
- OPTIONS (database 'db', collection 'warehouse');
+ SERVER "MongoDB server"
+ OPTIONS (
+ database 'db',
+ collection 'warehouse'
+ );
+```
--- Note: first column of the table must be "_id" of type "name".
+### Typical examples with [MongoDB][1]'s equivalent statements.
--- select from table
-SELECT * FROM warehouse WHERE warehouse_id = 1;
+#### `SELECT`
+```sql
+ SELECT *
+ FROM warehouse
+ WHERE warehouse_id = 1;
+```
+```
_id | warehouse_id | warehouse_name | warehouse_created
--------------------------+--------------+----------------+---------------------------
53720b1904864dc1f5a571a0 | 1 | UPS | 2014-12-12 12:42:10+05:30
(1 row)
-
+```
+```
db.warehouse.find
(
{
@@ -405,13 +393,14 @@ db.warehouse.find
"warehouse_name" : "UPS",
"warehouse_created" : ISODate("2014-12-12T07:12:10Z")
}
-
--- insert row in table
+```
+#### `INSERT`
+```sql
INSERT INTO warehouse VALUES (0, 2, 'Laptop', '2015-11-11T08:13:10Z');
-
-- Note: The given value for "_id" column will be ignored and allows MongoDB to
-- insert the unique value for the "_id" column.
-
+```
+```
db.warehouse.insert
(
{
@@ -420,20 +409,28 @@ db.warehouse.insert
"warehouse_created" : ISODate("2015-11-11T08:13:10Z")
}
)
-
--- delete row from table
-DELETE FROM warehouse WHERE warehouse_id = 2;
-
+```
+#### `DELETE`
+```sql
+DELETE
+ FROM warehouse
+ WHERE warehouse_id = 2;
+```
+```
db.warehouse.remove
(
{
"warehouse_id" : 2
}
)
-
--- update a row of table
-UPDATE warehouse SET warehouse_name = 'UPS_NEW' WHERE warehouse_id = 1;
-
+```
+#### `UPDATE`
+```sql
+UPDATE warehouse
+ SET warehouse_name = 'UPS_NEW'
+ WHERE warehouse_id = 1;
+```
+```
db.warehouse.update
(
{
@@ -445,39 +442,75 @@ db.warehouse.update
"warehouse_created" : ISODate("2014-12-12T07:12:10Z")
}
)
-
--- explain a table
+```
+#### `EXPLAIN`, `ANALYZE`
+```sql
EXPLAIN SELECT * FROM warehouse WHERE warehouse_id = 1;
+```
+```
QUERY PLAN
-----------------------------------------------------------------
Foreign Scan on warehouse (cost=0.00..0.00 rows=1000 width=84)
Filter: (warehouse_id = 1)
Foreign Namespace: db.warehouse
(3 rows)
-
--- collect data distribution statistics
+```
+```
ANALYZE warehouse;
-
```
Limitations
-----------
- * If the BSON document key contains uppercase letters or occurs within
- a nested document, `mongo_fdw` requires the corresponding column names
+ - If the BSON document key contains uppercase letters or occurs within
+ a nested document, ``mongo_fdw`` requires the corresponding column names
to be declared in double quotes.
- * Note that PostgreSQL limits column names to 63 characters by
+ - Note that PostgreSQL limits column names to 63 characters by
default. If you need column names that are longer, you can increase the
`NAMEDATALEN` constant in `src/include/pg_config_manual.h`, compile,
and re-install.
+Contributing and testing
+------------------------
-Contributing
-------------
Have a fix for a bug or an idea for a great new feature? Great! Check
out the contribution guidelines [here][3].
+### Running regression test.
+Run `mongodb_init.sh` file to load required collections.
+```sh
+source mongodb_init.sh
+```
+Finally, run regression.
+```sh
+make USE_PGXS=1 installcheck
+```
+However, make sure to set the `MONGO_HOST`, `MONGO_PORT`, `MONGO_USER_NAME`,
+and `MONGO_PWD` environment variables correctly. The default settings can
+be found in the `mongodb_init.sh` script.
+
+Useful links
+------------
+
+### Source code
+
+Reference FDW realisation, `postgres_fdw`
+ - https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD
+
+### General FDW Documentation
+
+ - https://www.postgresql.org/docs/current/ddl-foreign-data.html
+ - https://www.postgresql.org/docs/current/sql-createforeigndatawrapper.html
+ - https://www.postgresql.org/docs/current/sql-createforeigntable.html
+ - https://www.postgresql.org/docs/current/sql-importforeignschema.html
+ - https://www.postgresql.org/docs/current/fdwhandler.html
+ - https://www.postgresql.org/docs/current/postgres-fdw.html
+
+### Other FDWs
+
+ - https://wiki.postgresql.org/wiki/Fdw
+ - https://pgxn.org/tag/fdw/
Support
-------
@@ -486,7 +519,7 @@ PostgreSQL and EDB Postgres Advanced Server releases.
If you need commercial support, please contact the EnterpriseDB sales
team, or check whether your existing PostgreSQL support provider can
-also support mongo_fdw.
+also support `mongo_fdw`.
License
@@ -505,5 +538,3 @@ See the [`LICENSE`][4] file for full details.
[2]: https://github.com/enterprisedb/mongo_fdw/issues/new
[3]: CONTRIBUTING.md
[4]: LICENSE
-[5]: http://mongoc.org/libmongoc/1.17.3/installing.html#configuring-the-build
-[6]: https://github.com/json-c/json-c/tree/json-c-0.15-20200726#build-instructions--
From bef888f8d90ba81f2ecfb40f237f7c7bdefee97a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB?=
<41448637+mkgrgis@users.noreply.github.com>
Date: Thu, 5 Oct 2023 07:41:55 +0300
Subject: [PATCH 2/3] Fix spell errors, README.md
---
README.md | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/README.md b/README.md
index 816c823..d4d697a 100644
--- a/README.md
+++ b/README.md
@@ -43,7 +43,7 @@ query. This is a performance enhancement.
#### JOIN push-down
`mongo_fdw` now also supports join push-down. The joins between two
-foreign tables from the same remote MySQL server are pushed to a remote
+foreign tables from the same remote MongoDB server are pushed to a remote
server, instead of fetching all the rows for both the tables and
performing a join locally, thereby may enhance the performance. Currently,
joins involving only relational and arithmetic operators in join-clauses
@@ -81,8 +81,8 @@ network traffic between local PostgreSQL and remote MongoDB servers.
operation to the foreign server, instead of fetching rows from the
foreign server and performing the sort locally. Default is `true`.
* `mongo_fdw.enable_join_pushdown`: If `true`, pushes the join between two
- foreign tables from the same foreign server, instead of fetching all the
- rows for both the tables and performing a join locally. Default is `true`.
+ foreign tables from the same foreign server, instead of fetching all the
+ rows for both the tables and performing a join locally. Default is `true`.
* `mongo_fdw.enable_aggregate_pushdown`: If `true`, pushes aggregate
operations to the foreign server, instead of fetching rows from the
foreign server and performing the operations locally. Default is `true`.
@@ -265,7 +265,7 @@ All transformation rules and problems **yet not described**.
Generated columns
-----------------
-`mongo_fdw` does't provides support for PostgreSQL's generated
+`mongo_fdw` doesn't provides support for PostgreSQL's generated
columns (PostgreSQL 12+).
**Behaviour with generated columns yet not tested and not described**.
@@ -312,7 +312,7 @@ Once for a database you need, as PostgreSQL superuser.
### Create a foreign server with appropriate configuration:
-Once for a foreign datasource you need, as PostgreSQL superuser.
+Once for a foreign data source you need, as PostgreSQL superuser.
```sql
CREATE SERVER "MongoDB server"
@@ -325,7 +325,7 @@ Once for a foreign datasource you need, as PostgreSQL superuser.
### Grant usage on foreign server to normal user in PostgreSQL:
-Once for a normal user (non-superuser) in PostgreSQL, as PostgreSQL superuser. It is a good idea to use a superuser only where really necessary, so let's allow a normal user to use the foreign server (this is not required for the example to work, but it's secirity recomedation).
+Once for a normal user (non-superuser) in PostgreSQL, as PostgreSQL superuser. It is a good idea to use a superuser only where really necessary, so let's allow a normal user to use the foreign server (this is not required for the example to work, but it's security recommendation).
```sql
GRANT USAGE ON FOREIGN SERVER "MongoDB server" TO pguser;
@@ -347,7 +347,7 @@ Create an appropriate user mapping:
Where `pguser` is a sample user for works with foreign server (and foreign tables).
### Create foreign table
-All `CREATE FOREIGN TABLE` SQL commands can be executed as a normal PostgreSQL user if there were correct `GRANT USAGE ON FOREIGN SERVER`. No need PostgreSQL supersuer for secirity reasons but also works with PostgreSQL supersuer.
+All `CREATE FOREIGN TABLE` SQL commands can be executed as a normal PostgreSQL user if there were correct `GRANT USAGE ON FOREIGN SERVER`. No need of PostgreSQL supersuer for security reasons but also works with PostgreSQL supersuer.
Create a foreign table referencing the MongoDB collection:
@@ -495,7 +495,7 @@ Useful links
### Source code
-Reference FDW realisation, `postgres_fdw`
+Reference FDW realization, `postgres_fdw`
- https://git.postgresql.org/gitweb/?p=postgresql.git;a=tree;f=contrib/postgres_fdw;hb=HEAD
### General FDW Documentation
From 9a785e5d273c21642624d260c0cf4559e9153065 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=D0=9C=D0=B8=D1=85=D0=B0=D0=B8=D0=BB?=
<41448637+mkgrgis@users.noreply.github.com>
Date: Thu, 5 Oct 2023 07:58:16 +0300
Subject: [PATCH 3/3] Fix spelling, README.md
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index d4d697a..081e225 100644
--- a/README.md
+++ b/README.md
@@ -227,11 +227,11 @@ command:
Similar to the server-level option, but can be
configured at table level as well.
-- **enable_aggregate_pushdown** as *boolean*, optional, default corresponding servel-lelel value
+- **enable_aggregate_pushdown** as *boolean*, optional, default corresponding server-level value
Similar to the server-level option, but can be configured at table level as well.
-- **enable_order_by_pushdown** as *boolean*, optional, default corresponding servel-lelel value
+- **enable_order_by_pushdown** as *boolean*, optional, default corresponding server-level value
Similar to the server-level option, but can be configured at table level as well.