Cloud Discovery

Amazon EC2 Discovery

CrateDB has native discovery support when running a cluster with Amazon Web Services (AWS). The discovery mechanism uses the Amazon EC2 API to to generate the list of hosts for the unicast host discovery (see Unicast Host Discovery).

There is a best practice how to configure and run a CrateDB cluster on Amazon EC2.

Microsoft Azure Discovery

CrateDB has native discovery support when running a cluster on Microsoft Azure infrastructure. The discovery mechanism uses the Azure Resource Management (ARM) API to generate the list of hosts for the unicast host discovery (see Unicast Host Discovery).

The discovery mechanism is implemented as a plugin and resides within the plugin folder of the CrateDB installation. However, since the Azure Java SDKs does have a lot of dependencies and we want to keep the plugin jar small in size the dependencies are not included in the CrateDB distribution.

Requirements

To make the plugin work you have to add the following Java libraries to the $CRATE_HOME/plugins/crate-azure-discovery folder:

activation-1.1.jar
adal4j-1.0.0.jar
azure-core-0.9.3.jar
azure-mgmt-compute-0.9.3.jar
azure-mgmt-network-0.9.3.jar
azure-mgmt-resources-0.9.3.jar
azure-mgmt-storage-0.9.3.jar
azure-mgmt-utility-0.9.3.jar
bcprov-jdk15on-1.51.jar
commons-io-2.4.jar
commons-lang-2.6.jar
commons-lang3-3.3.1.jar
gson-2.2.4.jar
jackson-core-asl-1.9.2.jar
jackson-jaxrs-1.9.2.jar
jackson-mapper-asl-1.9.2.jar
jackson-xc-1.9.2.jar
javax.inject-1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcip-annotations-1.0.jar
jersey-client-1.13.jar
jersey-core-1.13.jar
jersey-json-1.13.jar
jettison-1.1.jar
json-smart-1.1.1.jar
lang-tag-1.4.jar
mail-1.4.7.jar
nimbus-jose-jwt-3.1.2.jar
oauth2-oidc-sdk-4.5.jar
stax-api-1.0-2.jar

You can download the libraries using a simple build.gradle file:

apply plugin: "java"

repositories {
  jcenter()
}

dependencies {
    compile('com.microsoft.azure:azure-mgmt-utility:0.9.3') {
        exclude group: 'stax', module: 'stax-api'
        exclude group: 'org.slf4j', module: 'slf4j-api'
        exclude group: 'commons-logging', module: 'commons-logging'
        exclude group: 'commons-codec', module: 'commons-codec'
        exclude group: 'com.fasterxml.jackson.core', module: 'jackson-core'
        exclude group: 'org.apache.httpcomponents', module: 'httpclient'
    }
}

task azureLibs (type: Copy, dependsOn: ['compileJava']) {
  from configurations.testRuntime
  into "libs"
}

Running gradle azureLibs will fetch the required jars and put them into the libs/ folder from where you can copy them into the plugin folder.

Basic Configuration

To enable Azure discovery simply change the discovery.zen.hosts_provider setting to azure:

discovery.zen.hosts_provider: azure

The discovery mechanism can discover CrateDB instances within the same vnet or the same subnet of the same resource group. By default it will the vnet.

You can change the behaviour using the discovery.azure.method setting:

discovery.azure.method: subnet

The used resource group also needs to be provided:

cloud.azure.management.resourcegroup.name: production

Authentication

The discovery plugin requires authentication as service principle. To do so, you have to create an Active Directory application with a password. We recommened to follow the AD Application Guide .

The configuration settings for authentication are as follows:

cloud.azure.management:
  subscription.id: my-id
  tenant.id: my-tenant
  app:
    id: my-app
    secret: my-secret

For a complete list of settings please refer to Discovery on Microsoft Azure.