terraform-azurerm-lake-loader-vmss

A Terraform module for saving Delta / Iceberg format files into Azure Blob Storage for direct querying by warehouses.

Usage

See the examples/default directory for a working usage example of this module.

Requirements

Name	Version
terraform	>= 1.0.0
azuread	>= 2.39.0
azurerm	>= 3.58.0

Providers

Name	Version
azuread	>= 2.39.0
azurerm	>= 3.58.0

Modules

Name	Source	Version
service	snowplow-devops/service-vmss/azurerm	0.1.1
telemetry	snowplow-devops/telemetry/snowplow	0.5.0

Resources

Name	Type
azuread_application.app_registration	resource
azuread_application_password.app_pasword	resource
azuread_service_principal.sp	resource
azurerm_eventhub_consumer_group.enriched_topic	resource
azurerm_network_security_group.nsg	resource
azurerm_network_security_rule.egress_tcp_443	resource
azurerm_network_security_rule.egress_tcp_80	resource
azurerm_network_security_rule.egress_udp_123	resource
azurerm_network_security_rule.ingress_tcp_22	resource
azurerm_role_assignment.sc_app_ra	resource
azuread_client_config.current	data source
azurerm_resource_group.rg	data source
azurerm_storage_container.sc	data source

Inputs

Name	Description	Type	Default	Required
bad_topic_kafka_password	Password for connection to Kafka cluster under PlainLoginModule (note: as default the EventHubs topic connection string for writing is expected)	string	n/a	yes
bad_topic_name	The name of the bad Event Hubs topic that the loader will push failures into	string	n/a	yes
enriched_topic_kafka_password	Password for connection to Kafka cluster under PlainLoginModule (note: as default the EventHubs topic connection string for reading is expected)	string	n/a	yes
enriched_topic_name	The name of the enriched Event Hubs topic that the loader will pull data from	string	n/a	yes
kafka_brokers	The brokers to configure for access to the Kafka Cluster (note: as default the EventHubs namespace broker)	string	n/a	yes
name	A name which will be pre-pended to the resources created	string	n/a	yes
resource_group_name	The name of the resource group to deploy the service into	string	n/a	yes
ssh_public_key	The SSH public key attached for access to the servers	string	n/a	yes
storage_account_name	Name of the output storage account	string	n/a	yes
storage_container_name	Name of the output storage container	string	n/a	yes
subnet_id	The subnet id to deploy the service into	string	n/a	yes
accept_limited_use_license	Acceptance of the SLULA terms (https://docs.snowplow.io/limited-use-license-1.0/)	bool	false	no
app_version	Lake Loader app version to use. This variable facilitates dev flow, the modules may not work with anything other than the default value.	string	"0.1.4"	no
associate_public_ip_address	Whether to assign a public ip address to this instance	bool	true	no
bad_topic_kafka_username	Username for connection to Kafka cluster under PlainLoginModule (default: '$ConnectionString' which is used for EventHubs)	string	"$ConnectionString"	no
custom_iglu_resolvers	The custom Iglu Resolvers that will be used by Enrichment to resolve and validate events	list(object({ name = string priority = number uri = string api_key = string vendor_prefixes = list(string) }))	[]	no
default_iglu_resolvers	The default Iglu Resolvers that will be used by Enrichment to resolve and validate events	list(object({ name = string priority = number uri = string api_key = string vendor_prefixes = list(string) }))	[ { "api_key": "", "name": "Iglu Central", "priority": 10, "uri": "http://iglucentral.com", "vendor_prefixes": [] }, { "api_key": "", "name": "Iglu Central - Mirror 01", "priority": 20, "uri": "http://mirror01.iglucentral.com", "vendor_prefixes": [] } ]	no
eh_namespace_name	The name of the Event Hubs namespace (note: if you are not using EventHubs leave this blank)	string	""	no
enriched_topic_kafka_username	Username for connection to Kafka cluster under PlainLoginModule (default: '$ConnectionString' which is used for EventHubs)	string	"$ConnectionString"	no
java_opts	Custom JAVA Options	string	"-XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75"	no
kafka_source	The source providing the Kafka connectivity (def: azure_event_hubs)	string	"azure_event_hubs"	no
ssh_ip_allowlist	The comma-seperated list of CIDR ranges to allow SSH traffic from	list(string)	[ "0.0.0.0/0" ]	no
storage_container_path	Path within the storage container to write output into	string	"events/"	no
tags	The tags to append to this resource	map(string)	{}	no
telemetry_enabled	Whether or not to send telemetry information back to Snowplow Analytics Ltd	bool	true	no
user_provided_id	An optional unique identifier to identify the telemetry events emitted by this stack	string	""	no
vm_instance_count	The instance count to use	number	1	no
vm_sku	The instance type to use	string	"Standard_B2s"	no

Outputs

Name	Description
nsg_id	ID of the network security group attached to the Lake Loader Server nodes
vmss_id	ID of the VM scale-set

Copyright and license

Copyright 2023-present Snowplow Analytics Ltd.

Licensed under the Snowplow Limited Use License Agreement. (If you are uncertain how it applies to your use case, check our answers to frequently asked questions.)