DNS, Monitoring, Networking, Cloud Firewall, Cloud Database, interworks.cloud Azure Pack Cloud, Acronis Backup Cloud on Microsoft Azure, Acronis Backup Cloud on interworks.cloud
interworks.cloud Primary Data Center, Microsoft Azure - West Europe
February 17, 2023 10:45 EET
[Resolved] All services are operating normally. The Azure Pack tenant portal has been successfully redeployed and serves requests since November 9, 2022. A few users still remain pending to be granted access to their tenant resources. This process is expected to be completed by the end of March 2023.
October 14, 2022 09:08 EEST
[Monitoring] We have been closely monitoring all recovered services for the past weeks and there are no unexpected issues encountered, all services are operating within established parameters. The Azure Pack tenant portal is currently being redeployed and we expect the service to be back online by the end of next week. The Hosted Exchange services were originally scheduled to be decommissioned by the end of 2022, however after thorough consideration of the incident findings and the fact that there are currently critical vulnerabilities (CVE-2022-41040 and CVE-2022-41802) for which no patch is yet released by the vendor, it has been decided not to restore operational status of these services. The process of extracting mailbox data and delivering the produced pst files to each mailbox user is already underway and notifications are already being sent to affected users with instructions regarding accessing and downloading the mailbox pst files. At current extraction rates, we expect the entire extraction process to be completed by November 15, 2022 at the latest.
A detailed incident report will follow in the upcoming week.
October 7, 2022 14:55 EEST
[Identified] Further to the previous announcement, any customers or partners who’d like to receive further information about the incident, despite the above conclusion, are welcome to request so via a simple email at dpo@interworks.cloud.
October 7, 2022 14:44 EEST
[Identified] Final notification of Interworks Single Member SA on the security incident of 28/08/2022
Following up on the conclusion of the Forensic Analysis report we received late night of 04/10/2022 by the two independent firms who investigated the incident and prepared the final report, we are glad to announce that there was absolutely no evidence of data exfiltration discovered (whether customer or company data), nor has there been any other data breach incident.
The however unfortunate security incident which caused great turbulence and frustration to partners and their customers and for which the company would like to extend its sincere apologies once more, was swiftly dealt with the outmost priority, transparency and in absolute cooperation with the Authorities. We would further like to emphasize that all actors who participated in the incident’s management, from Interworks’ personnel to its partners, resellers, suppliers and mostly—in their majority—customers, displayed exceptional composure, professionalism and, primarily, understanding in this difficult time of the company. The management and personnel of Interworks will never forget the support it received even from contributors who’d prefer to remain anonymous. Finally, we remain committed to learning from the many useful lessons the incident provided, further increasing the security of our networks and infrastructure.
September 20, 2022 16:02 EEST
[Identified] The forensic report for the incident of 28/08/2022 is expected to be completed and delivered next week.
Also, we’d like to inform you that we have recovered the Exchange servers and we are currently in the final process of verifying accessibility to customer data. We are also evaluating all possible technical options for the secure and compliant delivery of these past emails to our customers. More to follow on this procedure soon.
September 8, 2022 09:54 EEST
[Investigating] Network traffic has been successfully rerouted through an alternate provider and network communication between our Athens and Thessaloniki data centers have been restored. VM workloads should be accessible at this time. Our team is closely monitoring network operations to ensure users do not experience any unexpected behavior.
September 8, 2022 08:48 EEST
[Investigating] We are currently experiencing networking issues with one of our providers which affects the communication link between our Athens and Thessaloniki data centers. VM workloads currently hosted in Athens data center may experience network disconnections. Our engineers are actively working to reroute affected traffic from an alternate provider
September 5, 2022 12:42 EEST
[Investigating] Recovery of services and customer production workloads continues successfully, with still no reports of data corruption or loss.
Minor performance issues may be experienced (at a decreasing rate) which are unrelated to the incident but intermittently unavoidable to compensate for increased backup and recovery traffic.
We appreciate your cooperation in allowing our Partner/Customer success team to handle all tickets, albeit with some delays.
September 3, 2022 13:15 EEST
[Investigating] The issue with Acronis Backup Cloud on interworks.cloud has been successfully resolved and the service is fully operational at this time
September 3, 2022 11:29 EEST
[Investigating] A DNS related issue occurred today which affected the Acronis Backup Cloud on interworks.cloud service. Users may have experienced network timeout errors while trying to perform backup/restore operations. Our engineers have identified the cause and have taken all necessary actions to resolve it. The service will be fully operational again in approximately 3 hours.
The recovery process of virtual machines and production workloads still continues as planned and is not affected from the above issue.
Next regularly scheduled daily update: Sunday, 4/9/2022 at 12:00 noon
September 2, 2022 13:55 EEST
[Investigating] The recovery process of virtual machines and production workloads continues uninterrupted on a 24/7 basis. Customers whose VMs are now back online for more than 24 hours, also report stable, reliable and error-free operation (besides initial reports of data completeness).
For clarification purposes, we would like to inform you that all recovered workloads and backups are stored on and running from isolated and secured environments.
Next regularly scheduled daily update: Saturday 3/9/2022 at 12:00 noon. Interim updates may follow sooner, if needed.
September 1, 2022 16:57 EEST
[Investigating] The recovery process of virtual machines and production workloads is still in progress. So far, we have no reports of data corruption or losses by any customers who have recovered their data.
Α dedicated security team performs deep forensic investigation of the affected internal infrastructure and closely monitors all incoming and outgoing traffic according to our incident response procedure.
We have started deployment of a new isolated environment in order to facilitate and expedite the recovery to both primary and DR regions.
Next expected update: Friday 2/9/2022 at 12:00 noon.
August 31, 2022 13:50 EEST
[Investigating] There are no special updates to report at this time since our last update; progress continues as planned at the expected rate in recovering services and production workloads, infrastructure-wise.
Next expected update: Wednesday 31/8/2022 at 20:00.
August 31, 2022 02:23 EEST
[Investigating] Cloud Database and DNS services have been restored and are once again operational. In case you continue to experience any technical issues with regard to these specific services, please contact our support team immediately.
Next expected update: Wednesday 31/8/2022 at 12:00 noon
August 30, 2022 22:31 EEST
[Investigating] Following on our work towards restoring all services and customer data, we're currently working on restoring our Cloud Database and DNS services and we'll update this page as soon as they become operational.
We continue to gradually restore mission critical systems at a steady pace.
Rest assured that we fully appreciate your anxiety and concerns but we respectfully ask you to acknowledge the large amount of incoming requests at our support line... We hope that you appreciate that long delays in our responses, are neither typical of our past behavior, nor by choice, particularly when we have requests for time estimates and questions on issues that we have not fully investigated yet. Please continue to monitor status.intreworks.cloud for updates.
Finally, Data Protection Authorities have been notified about the incident, as as we are obligated to do so by law, within 72 hours from the incident's detection.
Next expected update: Wednesday 31/8/2022 at 12:00 noon. August 30, 2022 14:45 EEST
[Investigating] Customer VM indexing has been successfully completed, as originally estimated; preparations are already underway for gradual restoration of mission critical systems, to be followed by customer production VMs.
However, we cannot guarantee at this time that the Hosted Exchange email services will be successfully restored, and if so, investigation is expected to last 10 more days from today, with uncertain results. Hence, customers are encouraged to seek alternative email services (e.g. Office 365). We deeply apologize for this development.
Next expected update: Tuesday 30/8/2022 at 20:00.
August 29, 2022 20:02 EEST
[Investigating] As the indexing of customer backup data continues at a satisfactory rate (expected completion still remains as originally estimated for tomorrow), major other crucial components are gradually getting back online, such as the Acronis on interworks.cloud installation.
We remain hopeful that these initial positive steps will well continue into the night.
Next expected update: Wednesday 30/8/2022 at 12:00 noon.
August 29, 2022 12:13 EEST
[Investigating] While the 24/7 investigation and remediation efforts are still ongoing, we can now confirm that our Thessaloniki-based data center fell victim of a large-scale cyber attack as of yesterday, Sunday 28/8/2022 at 01:09 am.
As soon as the attack was detected (01:10 am) by our 24/7 security personnel, containment efforts and counter measures have been and are still being taken to protect customer assets and work towards full remediation.
Right now, it appears that the vast majority of backup customer data will begin being accessible again (alas, not online) in about 24 hrs (current estimate as of 11:45 am).
Again, our priority right now is having all customer data recovered and accessible again by that timeframe. Unfortunately, an estimate of rolling online production workloads (“live” status) can only be given once we have ascertained full access to customer data.
While this page will be updated as soon as more information becomes available, our support site and partner account managers are also available. We appreciate your understating in perhaps experiencing longer than usual responses.
Next expected update: Mon 29/8/2022 at 20:00.
August 28, 2022 22:18 EEST
[Investigating] The efforts of our internal and external teams will be continuing 24/7 towards the investigation and remediation of the incident affecting the services mentioned above.
At this time, no estimation on an expected resolution time can be safely made with any level of confidence to justify doing so. The focus of all of our resources, as expected, is towards this goal. We are fully aware of the impact to your business but, at this time, we can only assure you that all that can be done is being done. We truly appreciate your understanding, patience and support.
Please stay tuned to this page for any official updates; we will be sharing them as soon as they become available.
Next expected update: Monday 29/8/2022 12:00 noon EEST.
August 28, 2022 14:44 EEST
[Investigating] We are still investigating the incident affecting all of our services in the Thessaloniki based data center. All of our resources and leading external partners are working to remedy the situation as soon as possible.
Next expected update: Sunday 28/8/2022 at 22:00 EEST (if not earlier).
August 28, 2022 09:04 EEST
[Investigating] There is currently an ongoing security incident which has affected most of our cloud infrastructure services. As a result the following services are not currently operational:
- Public/Private Clouds (Virtual Machines, virtual Network and Storage services)
- Hosted Exchange services
- Cloud Databases
- Monitoring services
- DNS hosting services
Our team is actively engaged and is currently performing extensive scanning of the infrastructure in order to pinpoint the exact causes. Please expect an update at 12:00 EEST.
Thank you for your understanding and cooperation in this matter.
August 28, 2022 02:25 EEST
[Investigating] We are currently investigating an issue that affects all cloud services.