diff --git a/docs/data/README.mdx b/docs/data/README.mdx index 0b48afb61..a0fff61fe 100644 --- a/docs/data/README.mdx +++ b/docs/data/README.mdx @@ -10,7 +10,7 @@ This section will walk you through the differences between the various platforms - **[RPC](#rpc)** - live network gateway - **[Horizon](#horizon)** - API for network state data -- **Galexie** - exports raw ledger metadata files +- **[Galexie](#Galexie)** - exports raw ledger metadata files - **[Hubble](#hubble)** - analytics database for network data | Features | RPC | Horizon | Galexie | Hubble | @@ -70,3 +70,7 @@ Horizon is an API for accessing and interacting with the Stellar network data. I Horizon stores three types of data (current state, historical state, and derived state) in one database, and the data is available in real-time for transactional use, which makes Horizon more expensive and resource-intensive to operate. If you’re considering using Horizon over the RPC, let us know in the [Stellar Developer Discord](https://discord.gg/stellardev) or file an issue in the [RPC repo](https://github.com/stellar/soroban-rpc) and let us know why! You can [run your own instance of Horizon](./horizon/admin-guide/README.mdx) or use one of the publicly available Horizon services from [these infrastructure providers](./horizon/horizon-providers.mdx). + +## Galexie + +Galexie is a tool for exporting Stellar ledger metadata to Google Cloud Storage. diff --git a/docs/data/galexie/REAME.mdx b/docs/data/galexie/REAME.mdx new file mode 100644 index 000000000..a44954ff1 --- /dev/null +++ b/docs/data/galexie/REAME.mdx @@ -0,0 +1,39 @@ +--- +title: Galaxie Introduction +sidebar_position: 0 +--- + +## What is Galexie? + +Galexie is a tool for extracting, processing, exporting Stellar ledger metadata to external storage, and creating a data lake of pre-processed ledger metadata. Galaxy is the foundation of the Composable Data Pipeline (CDP) and serves as the first step in extracting raw Stellar ledger metadata and making it accessible. Learn more about CDP’s benefits and applications in this [blog post](https://stellar.org/blog/developers/composable-data-platform). + +## What Are the Key Features of Galexie? + +Galexie is designed to make streamlined and efficient export of ledger metadata via a simple user-friendly interface. Its key features include: + +- Exporting Stellar ledger metadata to cloud storage +- Configurable to export a specified range of ledgers or continuously stream new ledgers as they are created on the Stellar network +- Exporting ledger metadata in XDR which is Stellar Core’s native format. +- Compressing data before export to optimize storage efficiency in the data lake. + +**Galexie Architecture** + +![](/assets/galexie-architecture.png) + +## Why XDR Format? + +Exporting data in XDR—the native Stellar Core format—enables Galexie to preserve full transaction metadata, ensuring data integrity while keeping storage efficient. The XDR format maintains compatibility with all Stellar components, providing a solid foundation for applications that require consistent access to historical data. Refer to the [XDR](/docs/learn/encyclopedia/data-format/xdr) documentation for more information on this format. + +## Why Run Galexie? + +Galexie enables you to make a copy of Stellar ledger metadata over which you have complete control. Galexie can continuously sync your data lake with the latest ledger data freeing you up from tedious data ingestion and allowing you to focus on building customized applications that consume and analyze exported data. + +### What Can You Do with the Data Lake Created by Galexie? + +Once data is stored in the cloud, it becomes easily accessible for integration with modern data processing and analytics tools, enabling various workflows and insights. + +The pre-processed ledger data exported by Galexie can be utilized across various applications, such as: + +- Analytics Tools: Analyze trends over time. +- Audit Applications: Retrieve historical transaction data for auditing and compliance. +- Monitoring Systems: Create tools to track network metrics. diff --git a/docs/data/galexie/admin_guide/README.mdx b/docs/data/galexie/admin_guide/README.mdx new file mode 100644 index 000000000..71669ff03 --- /dev/null +++ b/docs/data/galexie/admin_guide/README.mdx @@ -0,0 +1,10 @@ +--- +title: Admin Guide +sidebar_position: 0 +--- + +import DocCardList from "@theme/DocCardList"; + +This guide provides step-by-step instructions on installing and running the Galexie. + + diff --git a/docs/data/galexie/admin_guide/configuring.mdx b/docs/data/galexie/admin_guide/configuring.mdx new file mode 100644 index 000000000..069497922 --- /dev/null +++ b/docs/data/galexie/admin_guide/configuring.mdx @@ -0,0 +1,46 @@ +--- +title: Configuring +sidebar_position: 20 +--- + +# Configuring + +## Steps to Configure Galexie + +1. **Copy the Sample Configuration**: Use the provided sample configuration file [config.example.toml](https://github.com/stellar/go/blob/master/services/galexie/config.example.toml). + +2. **Modify the Configuration File**: Rename the file to `config.toml` and update settings as needed. + +### Key settings include: + +#### GCS Bucket + +Specify the Google Cloud Storage (GCS) bucket where Galexie will export Stellar ledger data. Replace `destination_bucket_path` with the complete path of your GCS bucket, including any subpaths. + +Example: + +```toml +destination_bucket_path = "stellar-network-data/testnet" +``` + +#### Stellar Network + +Set the Stellar network for which you want to create the data lake. + +Example: + +```toml +network = "testnet" +``` + +#### Data Organization (Optional) + +Control how the exported data is organized in the GCS bucket. For example, the following configuration adds 64 consecutive ledgers per exported file, and creates a new directory for every 1000 files: + +```toml +# Number of ledgers stored in each file +ledgers_per_file = 64 + +# Number of files per partition/directory +files_per_partition = 1000 +``` diff --git a/docs/data/galexie/admin_guide/installing.mdx b/docs/data/galexie/admin_guide/installing.mdx new file mode 100644 index 000000000..05f31cf3e --- /dev/null +++ b/docs/data/galexie/admin_guide/installing.mdx @@ -0,0 +1,12 @@ +--- +title: Installing +sidebar_position: 30 +--- + +# Installing + +To install Galexie, retrieve the Docker image from the [Stellar Docker Hub registry](https://hub.docker.com/r/stellar/stellar-galexie) using the following command: + +```shell +docker pull stellar/stellar-galexie +``` diff --git a/docs/data/galexie/admin_guide/monitoring.mdx b/docs/data/galexie/admin_guide/monitoring.mdx new file mode 100644 index 000000000..0354af497 --- /dev/null +++ b/docs/data/galexie/admin_guide/monitoring.mdx @@ -0,0 +1,6 @@ +--- +title: Monitoring +sidebar_position: 50 +--- + +# Monitoring diff --git a/docs/data/galexie/admin_guide/prerequisites.mdx b/docs/data/galexie/admin_guide/prerequisites.mdx new file mode 100644 index 000000000..34a7a5889 --- /dev/null +++ b/docs/data/galexie/admin_guide/prerequisites.mdx @@ -0,0 +1,23 @@ +--- +title: Prerequisites +sidebar_position: 10 +--- + +# Prerequisites + +### 1. Google Cloud Platform (GCP) Account + +Galexie exports Stellar ledger metadata to Google Cloud Storage (GCS), so you need a GCP account with: + +- Permissions to create a new GCS bucket, or +- Access to an existing bucket with read/write permissions. + +### 2. Docker (Recommended) + +Galexie is available as a Docker image, which simplifies installation and setup. Ensure you have Docker Engine installed on your system ([Docker installation guide](https://docs.docker.com/engine/install/)). + +> **_NOTE:_** While it is possible to natively install Galexie (without Docker), this requires manual dependency management and is recommended only for advanced users.] + +### Hardware Requirements + +The minimum hardware requirements for running Galexie are: diff --git a/docs/data/galexie/admin_guide/running.mdx b/docs/data/galexie/admin_guide/running.mdx new file mode 100644 index 000000000..56bcf666b --- /dev/null +++ b/docs/data/galexie/admin_guide/running.mdx @@ -0,0 +1,101 @@ +--- +title: Running +sidebar_position: 40 +--- + +# Running + +With the Docker image available and the configuration file set up, you're now ready to run Galexie and start exporting Stellar ledger data to the GCS bucket. + +The primary way of running Galexie is using the `append` command. + +## Append Command + +Using the `append` command, Galexie can either continuously scan the network for new ledgers and export them, or work on a fixed ledger range and stop when it is exported. + +Syntax: + +```shell +stellar-galexie append --start [--end ] [--config-file ] +``` + +Arguments: + +`--start ` **(required)** + +- The starting ledger sequence number of the range being exported. + +`--end ` **(optional)** + +- The ending ledger sequence number of the range being exported. If unspecified or set to 0, the exporter will continuously export new ledgers as they appear on the network. + +`--config-file ` **(optional)** + +- The path to the configuration file. If unspecified, the application will look for a file named `config.toml` in the current directory. + +Example usage: + +```shell +docker run --platform linux/amd64 -d \ +-v "$HOME/.config/gcloud/application_default_credentials.json":/.config/gcp/credentials.json:ro \ +-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json \ +-v ${PWD}/config.toml:/config.toml \ +stellar/stellar-galexie \ +append --start 350000 --end 450000 --config-file config.toml +``` + +`--platform linux/amd64` + +- Specifies the platform architecture (adjust if needed for your system). + +`-v` Mounts volumes to map your local GCP credentials and config.toml file to the container: + +- `$HOME/.config/gcloud/application_default_credentials.json`: Your local GCP credentials file. +- `${PWD}/config.toml`: Your local configuration file. + +`-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json` + +- Sets the environment variable for credentials within the container. + +`stellar/stellar-galexie` + +- The Docker image name. + +### Resumability: + +A useful feature of the `append` command is that it supports basic resumability. If Galexie is interrupted while it is exporting ledgers, on restart it scans for the first missing ledger after the specified start ledger. It then resumes exporting from the first missing it detects. Resumability is enabled by default, so when restarting Galexie just use the same start ledger as before, and Galexie will pick up where it left off. + +## Scan-and-fill Command + +One of the downsides of the `append` command is that it may miss gaps if there are multiple non-sequential gaps in the export range. If you suspect such gaps, use the `scan-and-fill` command. The `scan-and-fill` runs a lot slower than `append` but it comprehensively scans the specified range to detect and fill any missing ledgers, guaranteeing data completeness. Because of its slower speed of execution, this command should be used only when gaps in data are suspected. + +Syntax: + +```shell +stellar-galexie scan-and-fill --start --end [--config-file ] +``` + +Arguments: + +`--start ` **(required)** + +- The starting ledger sequence number of the range being exported. + +`--end ` **(required)** + +- The ending ledger sequence number of the range being exported. + +`--config-file ` **(optional)**: + +- The path to the configuration file. If unspecified, the exporter will look for a file named “config.toml” in the current directory. + +Example usage: + +```shell +docker run --platform linux/amd64 -d \ +-v "$HOME/.config/gcloud/application_default_credentials.json":/.config/gcp/credentials.json:ro \ +-e GOOGLE_APPLICATION_CREDENTIALS=/.config/gcp/credentials.json \ +-v ${PWD}/config.toml:/config.toml \ +stellar/stellar-galexie \ +scan-and-fill --start 6400 --end 6800 --config-file config.toml +``` diff --git a/docs/data/galexie/admin_guide/setup.mdx b/docs/data/galexie/admin_guide/setup.mdx new file mode 100644 index 000000000..9f0fdb885 --- /dev/null +++ b/docs/data/galexie/admin_guide/setup.mdx @@ -0,0 +1,22 @@ +--- +title: Setup +sidebar_position: 10 +--- + +# Setup + +## Set up your Google Cloud Platform (GCP) credentials + +Create application default credentials by using your user account for your GCP project by following these steps: + +1. Download the [SDK](https://cloud.google.com/sdk/docs/install). +2. Install and initialize the [gcloud CLI](https://cloud.google.com/sdk/docs/initializing). +3. Create [application default credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc#google-idp) and it should automatically store in this location: `$HOME/.config/gcloud/application_default_credentials.json.` +4. Verify that this file exists before moving on to the next step. + +## Create a Google Cloud Storage (GCS) bucket + +If you already have a GCS bucket with read and write permissions, you can skip this section. If not, follow these steps: + +1. Visit the GCP Console's Storage section (https://console.cloud.google.com/storage) and create a new bucket. +2. Choose a descriptive name for the bucket, such as `stellar-ledger-data`. Refer to [Google Cloud Storage Bucket Naming Guideline](https://cloud.google.com/storage/docs/buckets#naming) for bucket naming conventions. Note down the bucket name, you will need it later during the configuration process. diff --git a/static/assets/galexie-architecture.png b/static/assets/galexie-architecture.png new file mode 100644 index 000000000..00873711c Binary files /dev/null and b/static/assets/galexie-architecture.png differ