Getting Started Verato with Azure: Connectivity Guide

Verato Connectivity Guide via Azure

Azure is a comprehensive cloud platform that offers a wide range of services, including computing, analytics, storage, networking, databases, among others.

Azure provides the infrastructure for Microsoft cloud services, providing a vast ecosystem where different services, including Azure Data Factory (ADF), can interact and operate in an integrated manner.

Verato understands the importance of maintaining a consistent environment to get efficient and secure data pipelines processes. That's why we streamline the integration process by enabling the connectivity through Azure.

The following step-by-step guide describes the processes to establish a safe and reliable connection and data consuming between Verato and the customer's Azure Microsoft services.  

• Process of establishing connectivity to Verato via Azure:

  1. Upload public key cert to certificates in the function app setting.
  2. Create function app configuration that references the public cert path, using the following prefix: /var/ssl/certs/{thumbprint}.der
  3. Extract .keyfile using the .p12 certificate with the command: openssl packs12 -in <>.p12 -out <>.key -nodes - nocerts
  4. Store Private key in key vault as a secret
       a) Remove the "BEGIN RSA" line as well as the "END RSA" line
  5. Create function app config that references the private key in key vault 

•     def create_private_key():
        key = get_key()
        private_key_path = write_temp_file(key)
        return private_key_path
      def get_key():
        begin = '-----BEGIN RSA PRIVATE KEY-----\n'
        from_vault = str(os.environ['verato_key']).replace(' ', '\n')
        end = '\n-----END RSA PRIVATE KEY-----\n'
        key = begin + from_vault + end
        return key
      def write_temp_file(key):
        path = ""
        with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.pem') as temp_key_file:
            temp_key_file.write(key)
            path = temp_key_file.name
        return path
      public_cert_path = os.environ['verato_public_cert_path']

Azure Data Factory (ADF) Deployment Process within Verato

Azure Data Factory (ADF) is a data integration service designed for building, orchestrating, and automating data pipelines in the cloud.

This service allows to create data-driven workflows for orchestrating data movement and transformation at scale connecting to various data sources, transforming the data using data flow, and then loading it into different destinations. It's particularly useful for Extract, Transform, Load (ETL) processes.

ADF integrates with a wide range of Azure services as well as on-premises data sources and third-party services. 

The Azure Data Factory deployment process within Verato entails several key steps to ensure that data pipelines are correctly set up and functioning as expected.

• By following these steps, Azure Data Factory can securely manage and use the .p12 file to authenticate connections to external services.

  1. Verato requires a security certificate to be attached when using an HTTP linked service.
  2. Store the Verato provided .p12 file in Azure Key Vault.
  3. Create an HTTP linked service in ADF.
  4. Reference the .p12 file from Key Vault in the linked service configuration.
  5. Use this linked service in your pipeline activities to make authenticated API calls.
  6. Repeat steps 1-5 after each new deployment code.