Lead Systems Architect-Engineer

Overview GovCIO is excited to announce an opening for a Lead Systems Architect-Engineer of Infrastructure Operations to support our impactful Administrative Office of the US Courts NLS project. This vital role involves managing an extensive data environment that currently ingests an average of 18-20TB of logging data daily across 60 indexers located in 2 data centers. Enjoy the flexibility of a fully remote position within the United States. Responsibilities Design, implement, and manage the Splunk ecosystem including Splunk Core, Enterprise Security, IT Service Intelligence (ITSI), Phantom (SOAR), Splunk Cloud, and Multi-Site Index Clustering. Monitor overall Splunk health through the Monitoring Console, ensuring optimal functionality of all system components. Track and analyze indexing rates, license usage, and alert metrics to proactively identify performance issues. Oversee CPU, memory, and disk utilization across Splunk components for efficient resource management. Respond rapidly to health alerts and anomalies, ensuring high operational availability. Troubleshoot user-reported issues regarding access, search failures, or alert functionality. Resolve ingestion challenges, focusing on Universal Forwarders, Heavy Forwarders, and HEC endpoints. Address issues related to log integrity such as missing or duplicate logs and escalate complex parsing errors as needed. Validate new data source integrations to ensure accuracy in log processing. Support data source owners during onboarding processes, including forwarder setup and syslog configuration. Ensure visibility of data flow from sources to indexers for performance verification. Manage security credentials and tokens for various data inputs and integrations. Administer user access permissions and manage entitlement reviews. Assist end-users in optimizing SPL searches and creating effective reports and dashboards. Keep knowledge base documents up to date, providing valuable resources for common issues. Document and escalate system issues to the Engineering team with detailed evidence. Manage support cases with Splunk for critical issues impacting system performance. Monitor ITSI service health, focusing on KPIs and correlation searches. Troubleshoot ITSI-related performance issues, ensuring timely resolutions. Conduct capacity management, monitoring index growth and retention policies. Perform routine system maintenance, cleaning up orphaned objects, and reviewing knowledge objects. Ensure compliance with data governance policies and maintain secure configurations. Take part in disaster recovery tests to confirm data integrity and backup processes. Document incidents and root cause analyses for future reference. Work closely with the Engineering team for escalations and system improvements. Qualifications Bachelor's degree with 10 years of relevant experience or a Master’s degree in a related field with 7 years of experience. Required Skills and Experience Deep expertise in Enterprise Security, ITSI, SOAR, and the Splunk product suite. Proficient in designing and deploying Splunk infrastructure environments. Clearance Required: Must be eligible to obtain and maintain AOUSC Public Trust. Company Overview At GovCIO, we are a team of innovators dedicated to revolutionizing government IT. Our commitment to delivering superior IT services enhances how government agencies operate and serve the public If you're passionate about making a difference and thrive in a collaborative environment, GovCIO welcomes you to join our transformation journey. What You Can Expect Interview & Hiring Process Virtual video interviews with the hiring manager and team. Camera-on policy during interviews. Photo ID verification is required. Hiring Process: Enhanced biometric ID verification and comprehensive background checks including criminal history, education, and employment history. Employee Perks Join us to enjoy a plethora of benefits such as: Employee Assistance Program (EAP). Corporate Discounts. Access to Learning & Development programs. Training and Certification Assistance. Referral Bonus Program. Internal Mobility Opportunities. Pet Insurance. Flexible Work Environment. We value the unique talents of our team members, which fuels our mission to support our customers and enhance our company culture. We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, disability, or status as a protected veteran. Posted Salary Range The anticipated salary for this position ranges from USD $105,000.00 - USD $145,000.00 per year, depending on various factors including experience and education.

Created: 2026-03-04