United Kingdom
The client is a B2B software company with a voice recognition platform utilizing AI capabilities. The product was designed to be hosted both on the company’s and its customers’ premises. It required a solid and consolidated installation script of both the client part of the application (backend and frontend) and the processing part (workers with Machine Learning tasks), as well as a set of sidecar services (database, search engine, etc.). They needed a vendor to introduce a set of tools that would smooth the operation and make the least possible steps and time to market. Ideally, the new approach should be simple enough to allow for handing over the product installation to the end customer’s engineers.
The stack consisted of a Drupal-based UI and a Drupal and Tomcat-based APIs as the application client, MariaDB as database engine, Sphinxsearch as full text search engine and a set of Python workers with worker-specific models. The later stages of the product roadmap required supporting a distributed installation approach where each of the application components was able to be deployed on a separate server and included in the lookup configs of all dependent services.
Since the client had an existing installer they wanted to use it as the foundation for the new one. It consisted of a set of Bash scripts and semi-related Ansible playbooks that had to be executed in a specific order. The client had to install the software on the customer’s premises in mostly manual fashion since the procedure was excessively complicated, and had bugs that could break the installation due to unexpected errors. That made every installation a unique experience for every customer.
The installer was requested to support any Linux (with focus on RPM and Debian-based distributions).
Wasted effort. Client’s developers had to be involved in the per-customer installation sessions, often for more than a day, instead of working on the product features and improvements.
A complicated definition of the existing toolkit. The installer had never been designed as a product but rather a mix of one-shot solutions to specific steps of the installation process that got convoluted over time. There was no consolidated configuration method for the installation, so the variables were set sporadically and unstructured. They were often set in the installer code directly for each customer individually which was cluttering the server's configuration entry and making it impossible to validate it after the initial setup.
Flexibility: the existing installer had several presets that allowed software and server customization to a certain degree, but did not provide the expected level of flexibility. The presets weren’t reliable to use either due to their sporadic development that was driven by customer setup customization inheritance rather than a structured approach. What’s worse, the client’s engineering team did not have a unified vision as to which components had to allow customization so the initial requirements were quite vague.
Lack of consistency and, as a result, reproducibility. The application components were partially containerized. The stack containing both containerized and on-host components lacked stability and reproducibility in its behavior across different installations.
Long time to market. All the aforementioned challenges were causing unnecessary troubleshooting and long debugging sessions. This was often delaying the delivery of the product to the end customers.
High barrier of entry. The current installer expected the engineer to have domain-specific knowledge and engineering expertise to put the setup together. On top of that, fine tuning the configuration was mandatory for almost every customer to match their hardware specifications. Combined together, this made it impossible for a customer’s engineer to quickly deploy the product on their premises.
We performed an initial audit to get familiar with the application stack and installer and identify issues and bottlenecks in the existing solution. We also analyzed the roadmap of both the installer and the product to consider future improvements.
For the initial implementation we aimed at stabilizing the existing installer behavior and its logic. Based on the audit results and the roadmap we suggested the following:
Make the application components' behavior expected and reproducible to decrease engineering effort required for their deployment.
Rework the existing installer by moving its logic definition to Ansible. Provide flexible and extendable Ansible Playbooks and application components’ templates to install the application on customer’s premises from scratch. This would make the end result use a more declarative approach.
As planned, we containerized all application components, created Docker build recipes, configuration and Compose templates for them. All configuration entries for the installer were consolidated in a single user-friendly YAML file that was easily customizable. This ensured centralized configuration management with Ansible and flexible application component distribution.
The next step was to provide a distributed installation of the application components on different servers, as well as automatic routing between the components on different servers. Since the product installation and configuration were handled during the first phase, our focus was mostly on updating the services grouping and distribution conditions to determine delivery to the specified hosts. We updated the templates to automatically fill the service configuration from the specified groups to allow cross-server communication of the components. The installer was extended with an extra step in the beginning to determine the OS distribution on-the-fly so that OS-specific components can be installed, providing support for a wide range of Linux distribution families.
As a result of our work:
All the components' packaging was consolidated which automated the client's product development and release workflows with a CI/CD platform.
The use of containerized application components brought stable and predictable behavior among all the servers and environments.
The Ansible playbook allowed the installer to run from a single place with a single command and deliver all the expected application components within a single run. This reduced engineering involvement in the process both due to the simplicity of the solution and its stability. Installation itself could be finished now within an hour (excluding the time to download the models to customer instances) instead of several days, the majority of which previously were interactive troubleshooting sessions.
Cluttered distribution installation and services configuration were resolved with two files: the first one listed on which host to install which component, the second one - a user-friendly YAML config file that defined the full configuration at the application level
The installer usage is fully documented and can be provided to the end customers without worrying about confusion in terms of the installer configuration or failed installations.
Have a need for a reliable distribution system? Corewide DevOps engineers can build one fine-tuned for your product!