/
Configuration Architecture

Configuration Architecture

Sept 04, 2018

The Configuration platform will be tightly integrated with the devops tooling (to faciliate rapid delivery). The API surface will enable a system admin to:

  1. Enumerate all the tenants for a given installation (top level system admin)
  2. Enumerate the high level topology for an installation (tenant system admin). That topology to include:
    1. Number of collector groups and their top level configuration (e.g. which integration, key settings, which message platform clusters)
    2. The Core Services service instances (a service is a logical grouping of microservices)
    3. The media api instances
  3. Add new collectors, service instances (subject to licensing restrictions)
  4. Remove service instances
  5. Configure each of these at a high level (integraitons, feature toggles, debug level)
  6. Connect service instances together e.g a NATS message service to a set of Consumers and in turn the consumers to an EventStore cluster and so on.

Most low level configuration values are generated at deployment time and held in the downstream devop platform data stores (E.g. Consul, Vault, Helm Chart store, Kubernetes configuration.

When a configuration change occurs the platform serialises the configuration for the installation into a Hashicorp Configuration Language RBV Config file (RBV Config HCL - alternatives considered were JSON, Yaml, XML and JSonnet - but HCL is neatest). This file is then passed to the Core Configuration Engine

Key Components

  • Configuration Management Subsystem
  • Core Configuration Data Store
  • Core Configuration Engine

Configuration Management Subsystem

This describes the overall system definition and topology, taking into account licensing, the required integrations and capabilities of the various subsystems/services needed for the system to execute.It can then output this data as a single file representation that can be processed by the Core Configuration Engine.

Key Features

Holds its data in a dedicated lightweight data store that can then be read and updated as needed. This supports the capability to describe the relationships and dependencies between various subcomponents.

It will NOT concern itself with enumerating and managing individual microservices but deal with a "Sub-system" abstraction layer. (the word "subsystem" is used to differentiate it from other abstractions across the architecture e.g. features - a licensing abstraction and services - a kubernetes abstraction). A Subsystem comprises of a set of services (mostly equatable to Kunernetes services) that are either core to the operation of the platform or support features and various pieces of functionality.  

Subsystems

This is an initial cut of the breakdown

  1. Capture Engines comprising of
    1. Collector Groups
    2. NATS streaming
    3. Consumer microservices
  2. Unified Storage:
    1. EventStore
    2. Minio
    3. ElasticSearch
  3. Recording Engine
    1. In progesss subscribers - metadata, media
    2. CallEnd Subscribers/Processors
      1. File Composer
      2. Callend Metadata
  4. Licensing and Multitenancy services
  5. In Call Features
    1. Record on Demand
    2. Suppression
  6. Call Replay APIs
    1. Metadata API
    2. Media API
    3. Annotations
  7. Identity Service
    1. Identity Server
    2. Identity Services Management APIs
  8. Export Services
  9. Import Services
  10. Health and monitoring services

Tasks/ features:

MMVP Features

  1. A break down of Microservices and services into the subsystems and then logical services as an approximate input for the definition of Kubernetes services. Initially this will be manually maintained in /wiki/spaces/RedboxHome/pages/2562762486 and then reflected in the chosen representation/schema (e.g. a YAML-based structure that describes the high level system definition  
  2. Storage of the overall Configration Management as a single file and simple API that supports GET of this file (alternative consider using GIT as the storage mechanism for storing the initial configuration management dataset as files) 
  3. A mechanism to trigger the Core Configuration Engine to deploy/redeploy

Subsequent work

  1. Break down the system defintion data strucutre into more of a schema of related objects so that more granular updates can be made 
  2. An API to perform CRUD, with EventSourcing/CQRS underpinning the writes
  3. The UI to render the architecture and support CRUD

Core Configuration Data Store

Guiding Principles

Microservice configuration: The challenge for configuration management is that we now have many many different standalone components to configure in a very dynamic hosting architecture.Each of those microservices/containerised services has different hosting and configuration needs.

Key/value pairs: At it's simplest Configuration is a collection of key/value (k/v) pairs. This collection of values for an overall platform would be extremely large so a way of subdividing/grouping those keys, but without losing the simplicity of the k/v approach. The k/v approach is intrinsic to the way configuration stores work in the containerisation domain. Furthermore all values have to be uniquely qualified to ensure they are not overwritten with the wrong value.

Shared values: consumers of configuration values often require the same values as other consumers therefore the keys for these values should be linked up in some way so that it is easy to ensure the right value is configured efficiently for all services that need it - possiubly by using hte same key names across different configuration (e.g. every component requires the EventStoreCluster.IPAddress and that is the name of the key in every appsettings.json file (which we could then read and determine the configuraiton needs of a microservice)

Evaluating Component Configuration Requirements: Need to define a process to register a microservice as part of the product, assert its configuration and hosting requirements and maintain/generate the necessary artefacts throughout the delivery lifecycle up to the point where it is spun up on a instance in the correct manner with the right configuration. Initially this will be manual, using the /wiki/spaces/RedboxHome/pages/2562762486 as a focal point but automated approaches will be considered.

Secrets: This form of configuration will be covered by our /wiki/spaces/RedboxHome/pages/2562776139 implementation - most secrets will not need to be externally accessible and will be generated within the platform

Generated Configuration Values: A large proportion of values will be generated by the orchestration engine (IP Addresses for access by other services/DNS names, passwords etc) - these can be written to and read from the kubernetes configuraiton layer or Consul.

Implementation

MMVP Features:

  1. Define a process, be it manual, automated or semi-automated to identify configuration entries that need to be maintained in Consul e.g. how does a service or subssytem advertise at development and then at deployment time 
  2. Implement Consul for configuration values that are supplied from outside Helm and Kubernetes - these can be retrieved from Consul at any point in the configuration lifecycle 
  3. Ensure that helm charts/config and k8s can work with Consul.
  4. Expose Consul API with simple security - CRUD requests  go straight to Consul
  5. Assess whether the Core Configuration Engine / Helm can be automatically triggered based on when a value in Consul has changed or simply explicitly trigger a redeploy.
  6. Scripts to extract configuration values

Stage 2:

  1. Wrap consul in an API with: our own contract definition and with our security
  2. Consider making it an EventSourcing/CQRS-based implementation (pros/cons tbd).
  3. A more granular fully automated mechanism by which updates to Consul trigger specific calls to the Core Configuration Engines

Core Configuration Engine

NB A provisional selection of Kubernetes as the container orchestration platform is inferred in this documentation 

The Core Configuration Engine compares the inbound RBV Config  file to the state held for the last deployment for that installation, and then works out how to apply the changes inferred. 

Config-To-Devops-Flow.vsdx


Implementation

MMVP:

  1. Define the initial structure for the config file and the code that can read it, validate it and write to it.
  2. Command line based implementation that accepts simple commands and the configuration file in its simplest form that can take - preferable integrated with the Command Line Interface (CLI)
  3. Integration with lcensing - cross match licensing with deployment to ensure only components that cover licensed features are deployed.
  4. Integrations with Devops tooling - Helm/Kubernetes
  5. Use CLI to output configuration values to manually supply to the Collector configuration process
  6. Call the  Media API management toolset.
  7. Basic rollback -e.g reapply previous configuration manually
  8. Call the engine from an API

Subsequent Work

  1. Fully drive this from an API with more granualr control over what the engine can do.
  2. Full rollback e.g. reapply last configuration
  3. Automated Collector Group management
  4. Integrate with expanding devops capabilities  such as full Blue/Green deployments



Sample RBV Config HCL File

This is a work in progress and is likely to be superceded by a YAML equivalent as YAML has better tooling support in .Net.

{ tenantId "7B12E957-371D-4D50-B092-B1EC308E9D1C" {
    environmentName = "Test1"
    license = "cn0i2eng0=eng0n-jv0rnvo 340gn24gnoiv ng2-jg09ejg/=" // base64 encoded license
    collectorGroups  [{
        groupId="23505945-75D7-4063-BADB-3578FA8FDC5D"
        collectorCount=2
        pairing="active-passive" // or "active-passive", needs count to be a multiple of 2.0
        integration {
            integrationId="0593FA49-B2B2-4ECE-B18D-DF4304EBC908"
            name="SIPREC-IPS1234"
            integrationSettings={ //integrationspecific settings}
            },
        collectorRuntime {
            //general collector settings
        }]
    services [{
        serviceName="NATSCluster1"
        serviceId="E02E8ECF-3ADD-49D6-A7AF-D13A0B3891CA"
        serviceType="NATSCluster"
        instanceCount="3"
        serviceConnections [{
            serviceType="SysLog"
            serviceId="3F9F89AB-AE54-4C13-9F2A-DE95CA032025"
        }] 
        // TODO: more NATS service config
    },
    {
        serviceName="RabbitMQCluster1"
        serviceId="9C06AD0B-FCEE-432F-BDA1-0272EC02BC20"
        serviceType="RabbitMQCluster"
        instanceCount="5"
        serviceConnections [{
            serviceType="SysLog"
            serviceId="3F9F89AB-AE54-4C13-9F2A-DE95CA032025"
        }]
        // TODO: more RabbitMQ service config
    },
    {
        serviceName="RabbitMQCluster1"
        serviceId="9C06AD0B-FCEE-432F-BDA1-0272EC02BC20"
        serviceType="RabbitMQCluster"
        instanceCount="5"
        serviceConnections [{
            serviceType="SysLog"
            serviceId="3F9F89AB-AE54-4C13-9F2A-DE95CA032025"
        }]
        // TODO: more RabbitMQ service config
    },
    {
        serviceName="MediaConsumers1"
        serviceId="9C06AD0B-FCEE-432F-BDA1-0272EC02BC20"
        serviceType="MediaConsumer"
        instanceCount="2"
        serviceConnections  [{
            serviceType="NATSCluster",
            serviceId="E02E8ECF-3ADD-49D6-A7AF-D13A0B3891CA"
            additionalParams {
                topicName="metadata"
            }
        },{
            serviceType="EventStoreCluster"
            serviceId="B3E37B18-1D05-40E6-93C1-F69947584040"
        },
        {
            serviceType="SysLog"
            serviceId="3F9F89AB-AE54-4C13-9F2A-DE95CA032025"
        }]
     
    },
     {
        serviceName="MetadataConsumers1"
        serviceId="9F26D3FA-623A-495D-800E-A2897B3921F9"
        serviceType="MetadataConsumer"
        initialInstanceCount="2"
        serviceConnections  [{
            serviceType="NATSCluster"
            serviceId="E02E8ECF-3ADD-49D6-A7AF-D13A0B3891CA"
            additionalParams={
                topicName="metadata"
            }
        },{
            serviceType="EventStoreCluster"
            serviceId="B3E37B18-1D05-40E6-93C1-F69947584040"
        },
        {
            serviceType="SysLog"
            serviceId="3F9F89AB-AE54-4C13-9F2A-DE95CA032025"
        }]    
    },
    { 
        serviceName="Syslog1"
        serviceId="3F9F89AB-AE54-4C13-9F2A-DE95CA032025"
        serviceType="SysLog"
        initialInstanceCount="2"
        serviceConnections  [         
        {
            serviceType="ElasticSearchCluster"
            serviceId="9CAB005E-D78C-4E57-8BF9-3D4B6E24B38A"
        }
        ]    
        
    },
        { 
        serviceName="ElasticSearchSyslogCluster"
        serviceId="9CAB005E-D78C-4E57-8BF9-3D4B6E24B38A"
        serviceType="ElasticSearch"
        initialInstanceCount="2"
        serviceConnections  [         
        {
            serviceType="ElasticSearchCluster"
            serviceId="9CAB005E-D78C-4E57-8BF9-3D4B6E24B38A"
        }
        ]    
        
    }
    }]
}

}


Configuration Tools

Helm - package manager for Kubernetes - allows applications to be defined and then deployed into Kubernetes, and updated.

Kubernetes - container orchestration engine

Consul - service enabling platform - can do service mesh/service discovery/key value pair repository for config

Vault - Secrets management engine - best uses consul as its storage engine

Collector Orchestration Engine - a yet to be specified set of artefacts to be able to deploy and provision/deprovision collectors - probably a bespoke piece of code

Media API Manager - a small amount of code and some utilities to configure IIS, create a website and deploy media api packages to, configure the api and also deproviison it - will use DISM, AppCmd and MSDeploy probably.


Add label