We’re writing to share additional and in-depth details surrounding our recent response to a cybersecurity matter. Our team is proud to share several positive efforts and initiatives that we believe protected the district, and data we manage, during our response. While we communicated as much as we could last week, we want to take an opportunity to ensure our staff and families understand the process guiding our response.
In the last two weeks, D20’s Information Technology team took a series of steps due to a vulnerability discovered by our firewall provider Palo Alto. This cybersecurity matter did not just impact D20, but many organizations across the nation. To maintain confidentiality and protect our layered security approach we cannot detail all actions, but we can share the following with you about our response in D20:
On Friday, April 12, clients utilizing Palo Alto were alerted to a vulnerability, but a solution (hot fix) was not immediately available.
That same day, D20’s IT teams took steps to disable one aspect of Palo Alto’s service, which was associated with the vulnerability.
On Tuesday, April 16, Palo Alto clients, including D20, were notified that a new hot fix was available, since it appeared to Palo Alto that the previous remediation did not fix the vulnerability.
D20’s IT teams put the hot fix into effect the same day.
On Wednesday, April 17, we reviewed our files with Palo Alto to determine if any indicators of compromise existed and were informed no such indications were observed.
From this point forward, the IT team executed additional and heightened security protocols, including monitoring network traffic consistent with known IP addresses of hackers associated with the vulnerability.
On Wednesday, April 24, while working on an unrelated hardware issue with Palo Alto, an indicator of compromise was detected, prompting a plan to address the compromise.
Immediately upon detection, the IT team engaged our cybersecurity partner, Mile High Cyber, to review findings with the Palo Alto team.
After confirming the indicator of compromise, the team instituted D20’s Cybersecurity Incident Response Plan.
As part of that plan, we convened a cyber incident response team, consisting of multiple departments from across D20. In our work, we reviewed existing plans and determined a temporary measure of taking down our network was appropriate.
While planning for the network outage, the team crafted communications to various audiences, including the broader leadership team, all staff and all families to alert them of the outage.
Plans were made to shift school on Thursday, April 25 to a paper-and-pencil learning day, as our systems would not allow for remote learning without a firewall in place. A two-hour delay allowed staff to pivot lesson plans in the morning. It was also determined that our physical security measures were operational, meaning it was safe to have students and staff in school. At no point was our district without the capability of communicating with families or emergency services as a result of the network outage.
As part of the response plan, we began the remediation process, which consisted of a series of steps to rebuild the firewall and implement security protocols.
On Thursday, April 25, the IT team continued working on the remediation process in partnership with Palo Alto and in coordination with Mile High Cyber.
While executing this work, one process in the Palo Alto system caused several hours of delays, leading the IT team to create alternative scenarios to re-launch the network.
After working past that process, the team resumed taking steps in the remediation process.
Out of an abundance of caution, we planned for another paper-and-pencil day on Friday, April 26.
On Friday, April 26, the network was restored after many hours of hard work by the IT team.
With the network back online, it still took several hours of work to re-issue security measures to all D20 programs and devices.
By the end of the day, most systems were back to normal, allowing for a normal school day on Monday, April 29.
From Saturday, April 27 and beyond, we maintained a posture of heightened awareness and monitoring, including additional steps to maintain a secure network. We set up a process to document and learn from this event, and will be working through that process in the near future.
As we communicated last week, the D20 IT team maintained an active investigation throughout this process to search for indicators of a data compromise, and to this date, no indication of compromise has been detected by our team or third-party vendors. Moving forward, we are partnering with an external agency to conduct a forensic review to verify our findings.
We want our broader community to know that amid our response, we took careful steps to ensure the safeguarding of data consistent with best cybersecurity practices, and while we understand a system-wide network outage caused disruption, we want to thank all members of our community for their patience.