Enabling SAIT to Reduce MTTR with Modern Observability

Siloed Visibility Impacting User Experiences

Disparate observability tools characterized SAIT's IT environment, each managed independently by different teams within the organization. This siloed approach hindered collaboration, scalability, and efficiency, making detecting and resolving issues promptly challenging, especially in heavy volume or load-inducing events like open enrollment. Student applications range from managing license keys for desktop applications to gathering data from machines in the labs. 

Manual configuration processes compounded these challenges, increasing operational overhead and delayed response times. SAIT recognized the need to consolidate its monitoring tools, streamline configuration processes, and enhance collaboration to improve overall observability and operational efficiency. At the time of engagement, SAIT’s observability team were seeking to gain actionable insights for issues or critical arising from:

• 110 hosts
• 100 million events per month
• 100 gig logs
• 20K synthetics

‍

TECHNICAL LANDSCAPE

Navigating Infrastructure Complexity

SAIT's primary concern lay in gaining visibility into its Ellucian Banner ERP systems, while infrastructure and network devices posed secondary concerns. The institution had minimal and siloed observability through  only a few tools like Dynatrace in place, with varying levels of coverage across different systems and departments, with each being monitored and managed individually. SQL servers lacked monitoring altogether, exacerbating visibility gaps. With a large amount of work being manually processed and no configuration management practices or know-how in place, the migration presented an additional perceived risk. 

Operating Systems:

• Redhat 7 Linux
• MS SQL

Hosts:

• Network/Domain/Proxy Hosts
• Infrastructure Hosts - Student Applications
  

Existing databases:

• Oracle databases
• MS SQL
  

Existing Logs:

• Tomcat logs and system logs
• Custom logs from apps that don’t have APM
  

Dashboards to be replicated in Datadog:

• Uptime robot 
• Nagios

Automating Operations for Proactive Monitoring

RapDev, a trusted partner with expertise in implementing monitoring solutions, collaborated closely with SAIT to understand their unique challenges and requirements. Leveraging their deep knowledge of Datadog and best practices in observability, RapDev proposed a comprehensive approach tailored to SAIT's specific needs.

The engagement began with a thorough assessment of SAIT's existing infrastructure, workflows, and pain points to identify opportunities for improvement. RapDev's team worked closely with SAIT's stakeholders to define key objectives, establish success criteria, and develop a roadmap for the deployment of Datadog.

“What's really exciting is we have built, with RapDev, an automated deployment platform so that when we want to get more licenses, change our existing agreement, or scale out to those 200 servers, we can do it in less than a week.” - Ross Henderson, Principal Consultant, SAIT

Empowering SAIT with Enhanced Visibility and Agility

Application Performance Monitoring

• Integration of Synthetic Testing: RapDev integrated synthetic testing capabilities into Datadog, allowing SAIT to proactively monitor application performance and user experience. Synthetic tests simulated real user interactions, enabling SAIT to identify performance bottlenecks and address issues before they impact end users.
• Visibility into Ellucian Banner ERP Systems: SAIT gained enhanced visibility into Banner application performance, with comprehensive monitoring from infrastructure to APM requests.
• Ansible Automation: Ansible automation was enabled for Windows and Linux on-premises development hosts, facilitating the installation of Datadog agents and APM components.
• Tagging Strategy/Standardization: Automated tagging was implemented across on-premises development hosts, enhancing organization and management.

Infrastructure

• Automated Deployment Platform: RapDev developed an automated deployment platform to streamline the configuration and provisioning of monitoring agents across SAIT's infrastructure. RapDev built a strong Ansible inventory for  SAIT that has facilitated scaling and reduced manual efforts.
• Collaborative Workflow: RapDev facilitated collaboration between SAIT's IT teams, providing guidance on best practices for monitoring configuration, alerting, and incident response. RapDev helped SAIT optimize its monitoring workflows and improve cross-team communication by promoting collaboration and knowledge sharing.
• Monitoring Rollout: RapDev implemented monitoring across infrastructure and application performance management (APM) using Ansible, with logging and synthetics.

Logging

• Logging Implementation: RapDev setup comprehensive logging capabilities, ensuring that SAIT could collect, search, and analyze log data efficiently. This setup provided SAIT with the necessary insights to troubleshoot and resolve issues faster.
• Service Level Objective (SLO) Use: While SLOs were not initially utilized, the groundwork was laid for future implementation. The robust logging setup provided a strong foundation for defining and monitoring SLOs in the future.
• Enhanced Support: Before RapDev's engagement, SAIT's monitoring relied primarily on Nagios alerts and lacked comprehensive visibility. RapDev's logging implementation drastically improved this, giving SAIT a unified view of its IT environment and enhancing support capabilities for internal resources.
• Integration with Existing Tools: RapDev ensured that the new logging setup was integrated with SAIT's existing tools like Terraform and Ansible, streamlining operations and providing a cohesive monitoring solution.

SAIT plans to leverage further Datadog capabilities to drive continual improvement and innovation in its operations. This includes expanding monitoring coverage, refining dashboards for deeper insights, and implementing advanced analytics for proactive issue resolution. By embracing a culture of innovation and collaboration, SAIT aims to stay at the forefront of technological advancements in the education sector.

RapDev's engagement with SAIT exemplifies the transformative power of comprehensive monitoring solutions in addressing operational challenges and driving business success. By deploying Datadog, SAIT has enhanced its visibility, efficiency, and support capabilities and is well-poised for future growth and innovation.