Panda Security Failure – 11 March 2015

Q) What happened?

On 11th March 2015 at around 12:00pm Panda Security pushed out a bad virus definition file to their virus protection software. The definition file was quickly retracted (about 60 minutes later) but during this time it incorrectly identified important system and program files as viruses, causing them to be “quarantined” and leading  to widespread system corruption and program failure

Panda Security have released a statement on the issue which can be found here:
http://www.pandasecurity.com/mediacenter/panda-security/information-regarding-issue-with-the-signature-file-pcop-retail-2015/

Q) Why did I have Panda Anti-Virus installed in the first place?

Panda has consistently been voted one of the best Anti-Virus programs on the market and won several industry awards. Because Panda runs “in the cloud” it is a very light weight application that runs on a wide range of PCs without disrupting performance.

Lennox IT have relied on Panda for several years as a first choice anti-virus solution, especially for older machines where performance can be an issue.

Q) What Lennox IT did

We became aware of the problem very soon after it arose and quickly worked at our own offices to diagnose the fault and develop a solution. Our fix – simply to load the computer into safe mode and use Panda’s own installer utility to replace damaged files – was tested on our own infected machines and then rolled out to clients over the next 24 hours.

At around 9:00pm on 11th March Panda Security also released their own repair application which we also rolled out to some clients on  Thursday 12th March.

Q) How come some PCs were completely broken and others had no problems at all?

The issue caused panda to incorrectly flag system files as viruses for approximately one hour however the damage largely depended on which files Panda was scanning at the time. In many cases Panda immediately attacked itself, destroying the software and stopping the problem at source. In other cases Panda attacked low level Windows/Systems files first and caused major system instability before finally turning on itself.

The extent of the damage to each PC depended on the following factors:

• If the PC was powered down or off-line then no damage will have been suffered
• If the PC was on but inactive then there was potential for minor damage to background services (software updaters, etc.)
• If the PC was in general use then there was potential for extensive damage
• If the PC was in use and the user attempted to fix the problem themselves or reboot then there was potential for extensive damage and operating system corruption

Q) Is my computer completely fixed now?

Yes and No. The fix we deployed (and the fix Panda themselves have supplied) restored most files back from quarantine and will repair the majority of problems. However in some cases simply copying the file back is not enough and sometimes system files cannot be copied back or their original location cannot be determined.

Until Panda release an alternate fix, we have done everything we can to restore your systems back to a fully healthy state and in 99% of cases we do not expect further problems as a result of this incident. However in some cases there will be lingering problems and we will work with you over the next few weeks to make sure that these are dealt with promptly.

In rare cases the PC will need to be rebuilt and Sam or I should have explained this to you when we removed your PC. We obviously have a bit of a back-log but we will endeavour to get these machines back to you as soon as we can from Monday 16th March.

Q) Have I lost anything I was working on? Is my data at risk?

No, absolutely not. Modern virus scanners tend to focus solely on applications not data so it was only program files that were affected. Any documents or files that you were working on during this time will be completely safe.

Q) Should I keep using Panda Anti-Virus?

They say that lightning doesn’t strike twice in the same place but obviously this has greatly undermined our trust in Panda Security and we have already switched some high priority machines over to an alternate virus scanner. Ultimately its your choice whether you want to continue with Panda or not and we can help you to install an alternate virus scanner if you wish. Panda is still the best choice for older machines but we would completely understand if you want to make the switch.

We are aware that many machines are currently running without a virus scanner following this incident and we will be around over the next week to fix this.

Q) What can we do to stop this happening again in the future?

Unfortunately there is nothing we can do to protect against something like this happening, hopefully it is such a high profile failure that all security companies will take note and put safe guards into their own procedures to prevent a repeat incident.

It does however demonstrate the importance of backups so if you don’t already have a adequate backup system in place then please speak with Sam or I and we can talk you through the different solutions we recommend. We will also be scheduling in an earlier-than-normal housekeeping check for those of you running our backup system so we can make sure everything is doing what it should.

Q) Any Questions?

If there’s anything else you want to ask then don’t hesitate to get in contact.

Kind Regards,

Lennox IT