Microsoft Update KB977165 triggering widespread BSOD

One of Microsoft's "Patch Tuesday" security fixes is triggering a widespread "Blue Screen of Death" problem.  The cause is not the update itself, but an existing infection.  So far, reports suggest that this problem affects Windows XP and Windows Vista. Once the update is applied and the system rebooted, Windows will bluescreen at boot.  When booted to Safe Mode, the system will freeze. Removing the update from the Windows Recovery Console or using live media will get the system booting again, at least until the update is reapplied. I have found that the root cause is an infection of %System32%\drivers\atapi.sys, and that replacing this file with a clean version will get the system booting normally. This is not the first time that an infection hitting atapi.sys has caused updates to trigger bluescreens.  If you are running Windows and have not yet applied this update, make sure you scan your computer thoroughly for infections before applying this update.  If you are experiencing this problem, get your computer to a professional that can replace the infected atapi.sys and clean any other malware from your computer.

References:

http://isc.sans.org/diary.html?storyid=8209

http://social.answers.microsoft.com/Forums/en-US/vistawu/thread/73cea559-ebbd-4274-96bc-e292b69f2fd1

Detailed Repair Instructions

Using the Windows XP Recovery Console

1. Boot from your Windows installation CD Insert your Windows installation CD and boot your computer. If your computer is not set to boot from CD first, you may need to reconfigure your BIOS or press a boot menu key (often F12, F8 or Esc). If you are unsure of how to do this, consult your favorite geek. As soon as the boot starts, you should see a message like "Press any key to boot from CD..." - press a key.

2. Start the Recovery Console After the CD loads (it may take a minute), you will be presented with a few choices. One of these options is to start a recovery by pressing "R". Press "R" to launch the Recovery Console.

* You may be asked to choose a Windows installation. If so, choose the damaged installation (probably "1").

* You may be prompted for the Administrator password. If you do not have one, press "Enter".

3. Identify your CD drive letter You should now be at the command prompt. Enter the following command:

map

Look for the drive letter for your CD drive. It may look something like this:

D:\Device\CdRom0

In this case, your CD drive is "D:".

4. Replace ATAPI.SYS Enter the following, replacing "D:" with your CD drive:

cd system32\drivers

ren atapi.sys atapi.old

expand D:\i386\atapi.sy_

You should see the message "1 file(s) expanded." - this indicates you have succeeded.

5. Reboot and scan for malware Reboot your computer. With a little luck, your computer will now boot normally. Because this problem is caused by malware, you should immediately scan your computer with up-to-date antivirus software.

UPDATE:

An atapi.sys infection may not be the only cause of this blue screen. While it does seem to be the most common cause, other infected drivers or drivers that make incorrect references to the updated kernel bits may also cause blue screens after this update is applied. Make sure you scan any computer with up-to-date antivirus software that can detect rootkits and check for updated drivers for your computer before applying this update.

UPDATE 2:

I have placed these instructions on my wiki. Any further changes will be posted there.

Comments

Nice bit of detective work Patrick!

I noticed your post over on MS's forum and followed your link here. BTW... the link you posted to the atapi.sys example hosted here immediately triggered my Avira anti-virus. Out of curiosity, iI uploaded my atapi.sys to virustotal and it came back clean (withone false positive) but didn't look anything like your example.

I believe you have hit the nail on the head. Hopefully word will get out about this post and people will save themselves some headaches by getting their machine cleaned BEFORE updating.

I've never had a digg account but I'm going to create one right now to digg this post. Hopefully it will help.

Again, great job!
Kind regards,
Ken

WOW - Makes for entertaining reading, has M$ not learnt anything by now, surely they must have test systems, even human operated ones !!!

It getting bad when a remote PC can BSOD your pc just for keeping it up to date, then paying a professional too fix it .... Hmmm shouldnt m$ be paying for it outta their pocket !!!!!

Luckily in this house, (and probably yours) we are the professionals !!!

PLZ HELP THE WORLD !!!

You just saved me a day of work and a lot of trouble!

Just a small addition:
You may want to replace a malicious atapi.sys file in the "\Windows\System32\dllcache" folder too.

Meanwhile MSRC confirmed that they are aware of potential BSOD occuring after KB977615 has been installed and are looking into the cause on their end (too).

Bye,
Freudi

Have encountered 2 XP PCs so far with BSOD fixed after uninstall of KB977615 using method mentioned HOWEVER I've checked the atapi.sys and both are clean and legitimate so the cause is definitely not ONLY attributed to an infected atapi.sys.

We had about 4 computers at the office crap out.

I've just encountered a laptop that wouldn't boot after the update. Same symptoms, failed in mup.sys in safe mode, etc etc.

Plugged the HD into my desktop and as soon as I touched atapi.sys trend screamed it was infected with trok_tdss.sme

Overwrote this file with another copy from a clean source and the laptop boots.

Just another addition: I can confirm what Chris says. The files were not infected, I was even able to reproduce this with a fresh install.

I am working on a customers PC that has the infected ATAPI.SYS file as I type. KB977165 installs new versions of several Kernel files and will most likely interact badly with several different infections - not just the TDSS Rootkit that infects ATAPI.SYS.
A full virus/malware scan of the infected hard disk using a donor PC or Boot CD will be required in most cases to identify the Rootkit infected system files.

RE:-
"Just another addition: I can confirm what Chris says. The files were not infected, I was even able to reproduce this with a fresh install."

This would imply that the latest version of atapi.sys is triggering the fault - notwithstanding the infected atapi.sys files out there. The latter have coincidentally been identified by the update process (i.e. a convenient red herring).

So the workaround is to replace the latest version of atapi.sys with the i386 compressed working version until MS rectify their latest version

Thanks I just got a computer in with that exact problem

GOOD WORK

Thanks, worked perfectly for me. User's computer had no virus protection and the blue screen. Your procedure immediately fixed the boot issue.

Thanks! That fixed my customer's BSOD immediately, and I think I have another one scheduled to come in . . .

Literally I have just had to re-install XPpro and all, repeat all the durrent updates and have not suffered any of the symptons described aboce

With the monthly updates Microsoft updates and run the Microsoft Virus Removal Tool as a stage in the update process. How come the utility doesn't clean Tdss (if it is present ?)

I had a machine come across my bench with this issue, first thing Wednesday morning. One of the first things I tried was running SFC form an ERD boot disk. it replaced several files including atapi.sys, but was still would not boot. only way to get the PC back up and running was to remove the patch.

Multiple scans, with no infection detected, and I tried re-installing the patch, only to get right back to Blue Screens.

In short, there is obviously more going on than just a problem with infected atapi.sys files.

I had the blue screen of death after installing the same patch KB977165 on my vista run PC. How to deal with that ??

Thanks

Very nice work Patrick,

We have seen this occur on a few machines at the FAA so I wrote a vbscript to loop through an .xls of machines and record the MD5 Checksum. Thought it may come in handy for yourself and some of your readers..

http://home.comcast.net/~jblizz/Atapi_MD5_Checker.zip

Be careful about saying "multiple" scans prove that computer is clean. Scans with what? Many have big problems with this type of infection, try Kaspersky TDSSkiller, Hitman Pro or Dr. Web Cureit. They are catching up all the time and some will see more than others. Think it is a major problem still.

Atapi.sys problem can be reproduced with right rootkit infection but there could be more, other files. "More going on" as someone said could be more rootkits, heh. One infection often leads to more. Or as was meant a combination with whatever MS bug, who knows.

The file I used was rather old btw. Many popular AVs did not recognize it at VirusTotal. If they would when fired off for real is another question. See no reason to have faith in them.

http://www.prevx.com/blog/139/Tdss-rootkit-silently-owns-the-net.html

Forgot link to Kaspersky tool http://support.kaspersky.com/viruses/solutions?qid=208280684 only for 32bit but does a bit more than sfc /scannow.

I have one question. When 1 of 46 identified on virustotal.com says Win32.Rootkit; does that mean eSafe sucks or the others suck?
Antivirus Version Last Update Result
a-squared4.5.0.502010.02.13-
AhnLab-V35.0.0.22010.02.12-
AntiVir7.9.1.1602010.02.12-
Antiy-AVL2.0.3.72010.02.13-
Authentium5.2.0.52010.02.13-
Avast4.8.1351.02010.02.12-
AVG9.0.0.7302010.02.12-
BitDefender7.22010.02.13-
CAT-QuickHeal10.002010.02.13-
ClamAV0.96.0.0-git2010.02.13-
Comodo39202010.02.13-
DrWeb5.0.1.122222010.02.13-
eSafe7.0.17.02010.02.11Win32.Rootkit
eTrust-Vet35.2.73002010.02.12-
F-Prot4.5.1.852010.02.12-
F-Secure9.0.15370.02010.02.13-
Fortinet4.0.14.02010.02.13-
GData192010.02.13-
IkarusT3.1.1.80.02010.02.13-
Jiangmin13.0.9002010.02.08-
K7AntiVirus7.10.9722010.02.12-
Kaspersky7.0.0.1252010.02.13-
McAfee58902010.02.12-
McAfee+Artemis58902010.02.12-
McAfee-GW-Edition6.8.52010.02.13-
Microsoft1.54062010.02.13-
NOD3248622010.02.12-
Norman6.04.082010.02.12-
nProtect2009.1.8.02010.02.13-
Panda10.0.2.22010.02.12-
PCTools7.0.3.52010.02.13-
Prevx3.02010.02.13-
Rising22.34.01.032010.02.11-
Sophos4.50.02010.02.13-
Sunbelt56752010.02.13-
Symantec20091.2.0.412010.02.13-
TheHacker6.5.1.4.1912010.02.13-
TrendMicro9.120.0.10042010.02.13-
VBA323.12.12.22010.02.12-
ViRobot2010.2.13.21862010.02.13-
VirusBuster5.0.21.02010.02.12-

Only 1 way to find out. Execute away. VT scan is nothing more than a hint as to what can be expected. VT logic/experience will say it must be false positive but my week old file was only detected by 13 out of 40 scanners - not going to bet with you. Which file are you talking about?, where did it come from?, what package is it part of? You don't get rootkit infection from a Firefox file straight from Mozilla. From a random RapidShare link? who knows... May be you just found yet another variant/generation.

Great article. Scary that none of the big names can stop this nasty thing.

There is a company here in Holland that claims they can remove the TDL3 rootkit. www.hitmanpro.com

Yes, and they update with references to this tdss crap. Being a removal tool they need to be on top of things. Why I said a "scan" does not have to mean much unless tool is hunting this down effectively. From what I know of tdss Dr. web Cureit, Kaspersky tdsskiller, Hitman Pro are good choices for "fixes". That trio was at least among the first to do something, pretty sure of that. Probably a very good idea to run several supplemental scans since 1 infection might in fact be 17. Then perhaps evaluate current protection setup or is it usage, heh. Something went wrong.

Fixing system critical files might not be a favorite among popular AVs ;) If everything does not go smoothly hell breaks lose.

I don't get it... the most surprising post for the week in planet fedora. Congrats!

PS: I use fedora (eh, and I admit sometimes Ubuntu).

To the people who say that scan results are coming up clean:

The TDSS variant which hooks ATAPI.SYS is smart enough to cloak itself in an active and booted system.

You must mount the drive from another operating system (either remove the drive and connect to a PC or boot from malware cleanup CD Rom)

I just thought I would throw that in there.

Just wanted to add my thanks to you for this article. Same issue here with rebotting constantly. REplace ATAPI.SYS and all is well. Running tdsskiller now and GMER to finish cleaning (along with MBAM, AVG, SB S&D, and then will pop drive out and scan from another machine with NOD32 on it).

When booting from the Windows CD there is no System32\drivers directory. Also, when you boot from cd you can't write to the make the changes to the atapi.sys file. How can this work with those limitation?

Thanks

A few additional notes:

* There are only a few rootkit detectors so far that can pick up this infection on a live system. The best way to check for this infection (or any rootkit) is to attach the hard drive to another computer and use that computer's scanners to scan the hard drive or use live media to perform the scans.

* eSafe is reporting a known-good atapi.sys as being infected. This is a false positive.

* I posted SHA1SUM results in a few other places. If you attempt to check the hashes using sha1sum.exe on a running, infected system, the rootkit will mask itself and return the "clean" hash. It will also mask itself from most other attempts at detection.

* Once you have found that you are infected with a rootkit, the only way to be sure you have gotten rid of all infections is to wipe out and reload your computer. If you have followed the steps above to get your computer booting again, it would be a good idea to back up your important data and reload your computer.

* If you have a netbook or are otherwise unable to boot from CD, you may be able to boot from a USB device or memory card. There are some (unsupported by Microsoft) ways to put Windows recovery systems on memory cards and USB memory sticks. It is also possible to use Linux live media. I will not provide the steps to do this, but you are welcome to look into these options. I actually used a Fedora live SD card to fix this problem the first time I encountered it on a client's Eee PC.

Anyone struggling against this problem should be able to get support from Microsoft, as outlined in their blog post on the subject:

http://blogs.technet.com/msrc/archive/2010/02/12/update-restart-issues-a...

Bootable Live-CDs:

Avira Rescue CD:
http://www.free-av.com/en/tools/12/avira_antivir_rescue_system.html

Dr. Web Live-CD:
http://www.freedrweb.com/livecd/

Bitdefender:
http://download.bitdefender.com/rescue_cd/

Kaspersky:
http://devbuilds.kaspersky-labs.com/devbuilds/RescueDisk/

Update first if tool does not to it automatically. They are all based on Linux but some use old version of everything so there can be problems booting. At least 1 will work unless computer is 15 years old.

Also legal to make a bootable usb with tools like Unetbootin or Shardana but I know at least Dr. Web has a known problem with that right now. Require a cd or something. CD is more foolproof.

I say legal because some brands like Symantec require a license to make such boot thingys. These are for public consumption :)

I got the same problem with Win7: no trouble until recent updates were installed, then windows got stuck at the boot up screen. Multiple system recoveries resulted in a working OS. None of the known malware or rootkit programs have an effect or even notice the problem, except for HitmanPro, which managed to remove atapi.sys at reboot. But there is this other file, jojep.sys, that appears to be unkillable. I still get an occasional BSOD. My Windows 7 cd does not allow for the work-around suggested here. Tips anyone?

Thank you for this information! It saved me from a lot of troubleshooting.

Here are the steps I took:

1) A full image backup of the drive with Macrium Reflect
2) A full scan with NIS2010
3) Backup of registry files and replaced with ones from Repair folder
4) A quick Spinrite 6 run and Memtest86+ for the memory
5) Replaced the SO-DIMM with another because it was flaky
6) Another scan this time ATAPI.SYS only, NIS found it infected
7) Replaced the ATAPI.SYS with a clean one
8) Booted the laptop successfully (only 15 processes with a clean registry!!)
9) sfc /scannow and Blacklight
10) Replaced the registry files so I could check the software

Tip: Always clean drives 'offline' if possible. Attach it to another PC with a USB-adapter or similar solution. It's easier to backup the important files, scan the whole drive and replace infected files.

I have got the blue screen problem. Attempting fix mentioned at beginning of article but cant figure out how to get to recovery step. Please help.

Reply to Mike:

I think you are making the same mistake that I did on the first try. I'm a novice to command prompt also.

You probably typed (cd system32\drivers) to the cd drive. You want to do it on your C: drive.

So, I will edit the instructions a bit, for novices.

1. C:
Enter

cd system32\drivers
Enter

ren atapi.sys atapi.old
Enter

(you replace the D: with whichever letter your cd drive is assigned)
expand D:\i386\atapi.sy_
Enter

You should see the message “1 file(s) expanded.” – this indicates you have succeeded.

Restart.

I hope this helps.

Sorry, my reply was to Will. I don't know where I got Mike from.

Problem solved on Asus EePC 1000HD.

pulled hd from asus and ran mcafee scan. that fixed the atapi.sys file and categorized it as a patch-sysfile.b trojan. however this still caused the bsod. so i replaced the atapi.sys from an OEM xp SP2 cd and that allowed the system to boot.

Thanks for this blog! Help a lot!

You guys are funny. I have been fighting this bug since late last year.

Tools used:
WinPE with a clean copy of atapi.sys
Combofix - run everytime...(only download from bleepingcomputer.com...all others are a hoax and make you pay for free utility)
ATF from Atribune - run everytime
SUPER AntiSpyware - sometimes to help clean left over bits
MalwareBytes Anti-Malware - sometimes to help clean other left over bits
Turn off System Restore...you should have done this first anyway
HiJackThis - sometimes if you get a browser hijacker with it
WinSockFix - if it messed up your network connections
Various other fixes for other trojans and viruses downloaded by the rootkit

Also check hosts file and lmhosts

This bug is a little bit worse then just a patch crapper. It downloads different versions of trojans and itself comes in different flavors. May also run in conjunction with atapi.sys.tmp.

Best case is replace atapi.sys, run combofix, and run a spyware zapper.
Worst case is the above along with all the other tools listed above and then end up rebuilding because it hosed up Windows files and registry keys and will take longer to fix then to rebuild.

Good luck...I have seen it over 150 times in last 3 months...I have over 4000+ users by the way...

Well, for this particular concern, you can call Microsoft PC Safety Department in a toll-free number 1.866.727.2338. They will assist you on this particular problem and just in case somthing went wrong, they can correct it instantly.

I notice Michael Bristow above (2010-02-12 at 11:48) said, "only way to get the PC back up and running was to remove the patch."

Any one else tried that?

Replacing ATAPI.SYS on Windows Server 2003 worked for me.

Removing the patch should work as well, however either way you're still inected and need to get cleaned.

I don;t want to hear this "crap" about malware/infections. I have a VERY clean environment and found that after the update of KB977165, it BSOD'd me and would not return. After I was able to uninstall KB977165, everything returned to normal and I still has a clean machine. Anything else on here/claims/fixes/whatever is MS rubbish.

BAD: KB977165 - uninstall or do not install
VIRUS: Still none found.

I've been dealing with a BSOD issue for about three weeks. It has really come to a head since patch Tuesday. Today I had four machines that exhibited symptoms that pointed to a MS security updates being the cause. I removed the update (KB977165) as prescribed on another web-site and got two of the four up and running. After reading this article I went about replacing a corupt of infected atapi.sys file on the two remaing systems. To my surprise both machines had NO atapi.sys file what so ever. That makes me think that some commonly used Antivirus or AntiSpyware programs must remove the infected files altogether. Once I replaced the missing atapi.sys file the remaining systems booted up normally.
Thanks for the help it is Greatly appreciated!

May be this will convince those who trust AVs http://www.youtube.com/user/markloman#g/u Even if they detect rootkit you can be screwed unless you use the few tools which actually work - alternatively a good live-cd. Almost guarantee there is more to remove/fix than atapi.sys, like what sfc will tell you, so a pure MS solution is not that good/safe.

Note he says in one of the first videos that Rootkit makers have now FIXED MS patch problem, heh. They have no interest in dead machines, only working zombies. So new updates from AVs required to prevent new infections.

Do any of these fixes apply to Windows Vista? So far I've found none for that OS in particular.

I had this happen to a co-worker of mine and I had to fix his system. atapi.sys was replaced but that didn't work. I also replaced iastor.sys and MBR. That seemed to do the trick. I was able to boot and update his AV and Malwarebytes. After a scan with Malwarebytes it found several instances of TDSS rootkit. Norton found nothing. Everything seems to be back to normal.

John - you said earlier:

"I don;t want to hear this “crap” about malware/infections. I have a VERY clean environment and found that after the update of KB977165, it BSOD’d me and would not return. After I was able to uninstall KB977165, everything returned to normal and I still has a clean machine. Anything else on here/claims/fixes/whatever is MS rubbish."

What are you using to determine that your system does not have this rootkit, and how are you using it ? (remove drive and scan on a diff PC, scan after system is booted, using a boot CD/USB drive)

Just curious - I have fixed 2 of 2 XP boxes that were BlueScreening by simply replacing atapi.sys

I also had this problem. I have only recovery disc for my laptop, so I used my wifes laptop as a donor of 3 files: atapi.sys, iastor.sys and ndis.sys, copied them to USB stick. I started infected laptop with Mandriva ONE live CD and replaced these files on my laptop. System works now. Finally solution was simply, but prepared a few days. Good Luck to all :)

I've tried using the boot CD to uninstall the updates through the command prompt but it hasn't been working. Does Vista have different commands for doing that?

Conclusion from MS http://blogs.technet.com/msrc/archive/2010/02/17/update-restart-issues-a... or use 64bit modern Windows ;)

Hitman Pro guy made this post http://www.wilderssecurity.com/showthread.php?t=265297 saying 75% of the cleaned computers had updated AV. Depressing but how it is.

Completely irritated with the stress caused by Microsoft! Thinks that Mr Bill Gates should also be paying for our shrink bills after this last update! I have just reformatted my laptop less than 48 hours ago, nothing, and I mean NOTHING has been infected as it has been doing nothing but re-loading software and files since it has been reformatted. Granted, it was not reformatted due to infection, it was due to the lameness that they call Vista, had been becoming more and more slow since day of purchase one year ago. I am a computer savy tech, just not completely in Vista, this is the first pc I have had to work on with Vista on it. Anyways, since the reformat, it has installed over 100 updates, last nite being the worst of them! As after it was finished went to a BLACK screen of death, yes I said black! When rebooted, it told me that there was no registry, then no hard drive! It gave no other options after several times of reboot, so I shut it down for the nite and tried again this morning. When started up this morning it gave the option of repairing system, then system restore back to normal. I am completely protected and none of my files are corrupted, but the fact remains.... Microsoft.. Why o Why, are you sending us to the looney bins with this crappy software update? I believe from now on, before ANY updates are done then I will research them first after, I hate to say it, but after everyone else updates and see if there are any massive flaws like this before I update anything. As in my line of work, I cannot be shut down for long periods of time like this. I have lost 12 hours of pc time, which means 12 hours of billable time, who do I bill? Bill Gates? Fat chance there!