Like all news stories, the VPNFilter router malware has now faded from the headlines. But the underlying problems are not going away and they are bad. Bigly bad. This is a detailed look at just how bad.

For starters, this was inevitable. The security of routers is disgraceful. As shown on the Bugs page of my RouterSecurity.org site, routers are buggy as heck. Most, if not all, the bugs listed there are security related. Worse still, router software/firmware is often quickly abandoned, meaning no firmware updates. Maybe someday, routers will ship with a "good until" date, but don't hold your breath. Even when there is new firmware to fix a bug, end users typically don't install the updates for a wide range of reasons. Inevitable, VPNFilter was.

According to Bruce Schneier, VPNFilter is "... a harbinger of the sorts of pervasive threats - from nation-states, criminals and hackers - that we should expect in coming years."

DETAILS

Is your router infected? None of your business.

The owner of a router has no way practical way to learn if their router is infected. Unlike simpler hacks, such as DNS server hijacking, there are no configuration settings in a router that we can look at or change. Some router hacks are given away by open TCP/IP ports, but not this one.

The second Cisco Talos writeup described the device destruction module in VPNFilter. If you can get a Linux command line interface to a router, and you knew the appropriate Linux commands, you could look for the the existence of the folders and processes described there as an indicator of a VPNFilter infection. For example, there should be a running process called VPNFilter. I have not seen this approach mentioned anywhere.

The report also mentioned that VPNFilter does some re-routing of traffic from port 80 to port 8888. Sometimes? All the time? If we test the router from the WAN side will be see port 8888 open? What about testing from the LAN side? Again, no one has offered specifics on what to look for or where to look.

One clear symptom should be the changing of HTTPS to HTTP requests as described in the second Talos report. But, this does not work all the time. This very site, for example, is HTTPS all the time. Even if you make an HTTP request to this site, it gets automatically changed by the webserver running the site to HTTPS. So, this site could not be used to test for a VPNFilter infection. We need tester websites and are not offered any.

Another thing the malware does is change any accept-encoding request header to not contain a gzip parameter. This too should be detectable, but no one says how.

The only detection approach mentioned in the press is to look at the network activity of a suspect router. This, however, means that the suspect router can not be directly connected to the Internet, it needs to be connected to another device, one capable of logging and/or monitoring the output of the suspect router. Good luck with that. Also, discussions that I have seen about scanning activity, fail to mention whether the give-away scans occur on the WAN side or the LAN side (or both).

Surprisingly, no one has published any symptoms of an infection that might be noticed on a device connected to an infected router. I think there are symptoms.

When the FBI filed their Affidavit with the court to take control of the toknowall dot com domain (used by VPNFilter), the story they presented to the court started with a person in Pittsburgh. Patient Zero, if you will, turned their router over the to FBI which found the VPNFilter malware installed on it. But, what made Patient Zero turn their router over to the FBI in the first place? The affidavit does not say. It is reasonable to assume that they noticed something unusual. What did they notice? Again, none of our business.

We also don't know how the infected routers were hacked.

Or, do we? Talos, which first went public with VPNFilter, said that new zero-day flaws were not involved and that all the known vulnerable devices "had well-known, public vulnerabilities." All four vendors, of devices known to be vulnerable, sounded very sure in their public statements that they knew which bugs were being exploited and that they had already issued fixes.

Which routers are vulnerable? We don't know.

Talos published a list of routers (and QNAP NAS devices) that they found hacked but they clearly said their list was incomplete. At the least, we have to assume that any and all Netgear and Linksys routers are vulnerable. QNAP specifically cited the vulnerable and patched versions of their QTS software. MikroTik was almost as clear, pointing out that firmware released after March 2017 should be safe.

That said, even if we had a complete list of known vulnerable devices today, nothing stops the bad guys from expanding the field by attacking other known flaws in other routers.
BINGO! Two days after I wrote this, we learned that many other routers are vulnerable. But, even that list is not complete according to Cisco Talos. As of June 8, 2018, I have seen four companies state that their routers are not vulnerable: Peplink (which also makes PepWave branded routers), Cisco, Eero and Juniper. Everyone else could be vulnerable, especially consumer routers. Surprisingly, pfSense, which brags about security, has said nothing (as of June 8, 2018).

The FBI got the word out to the media that they had taken control of a domain name used by VPNFilter. What they didn't brag about to the public was that they had not taken control of other servers. On June 6, 2018 we learned that VPNFilter is still sending data, it considers sensitive, to servers controlled by "the attackers."

Given all that, perhaps the worst aspect of VPNFilter is that the malware can brick a router.

What was a router yesterday becomes a paperweight today. This, as far as I know, is a new thing. Considering that over a half million routers were found to be infected, this could be a very big deal.

REMOVAL

Want to remove VPNFilter? Lotsa luck.

Most malware infections of routers can be removed simply by rebooting (or power cycling) the router. VPNFilter, however, is the second such malware known to persist across reboots. Yes, rebooting is a good thing, and it removes some of the malware, but enough remains to fully re-infect the router.

When VPNFilter can not phone home, it starts listening on all TCP (and UDP?) ports for a special Ethernet trigger packet. You can think of the trigger packet as a secret handshake. Trigger packets contain an IP address from which the remaining malware can download the rest of itself and fully re-infect the device. No doubt the creators of VPNFilter (thought to be Russians) know the IP addresses of the routers they have infected, so it would not be hard to re-infect them by sending trigger packets.

Not to mention the bug that let the Russians hack the router in the first place. Making it harder for them to re-infect the router, does nothing to block other bad guys from infecting the router with something else.

Because the FBI took control of a domain used by the malware, infected routers now phone home to the FBI when they boot up. So, in addition to the Russians being able to re-infect a device, so too can the FBI. US law probably does not allow the FBI to modify infected routers, but at least they know where the ones that have been rebooted are. In partnership with the Shadowserver Foundation, the location of known infected routers will be provided to ISPs and others that can notify the owners.

This certainly seems like a case where hacking back is called for. Perhaps another country, one where law enforcement can legally hack routers, will be persuaded to use the self-update mechanism of VPNFilter to fully remove the infection.

As for the rest of us, Talos suggested a factory reset, but I question this advice. For one thing, three of the four vendors of known vulnerable devices disagree. Only Linksys suggested doing a factory reset in their instructions to their customers. Netgear, MikroTik and QNAP did not say that it was necessary.

And, this brings up an interesting question - just what does a factory reset of a router do?

Does it install new firmware from a read-only copy? From a read-write copy? If it does install new firmware, how old is the firmware it restores and where did it come from? Or, does a factory reset simply reset the configuration options to their factory state and not modify the currently installed firmware at all? No one has addressed this. Its none of our business.

We do know that factory resetting a router, restores all the configuration options to their factory fresh state. But, why do this? If the VPNFilter malware changes some settings, we are not told which ones. Again, none of our business. If we knew what it changes, we could look out for those settings as a way to detect the presence of the malware. If the VPNFilter malware does not change any settings, then the public is being sold a placebo. Something that makes us feel good but actually does nothing.

Many routers let you backup the current configuration settings to a file. If the settings were backed up, then a router was reset to factory state, and then the settings were restored from the backup, have we done anything? None of our business.

All the vendors of the known vulnerable devices said to update the firmware and felt sure that the exploited bugs were fixed in the latest firmware. But, what if they are wrong? What if a device using the latest firmware was hacked because of a default password? Will a router let you install the current firmware on top of itself? Do they do sanity checks? Again, none of our business.

Sophos took a very detailed look at VPNFilter and concluded that we should power cycle, install the latest firmware and factory reset the device. They were the only one I ran across that suggested this triple play.

BIG PICTURE

Taking a step back, what is not said, anywhere, is that the infected routers are, frankly, crappy.

That is, they were developed with no thought to security. On my RouterSecurity.org site, I argue against using a router from your ISP or any consumer router for that matter. And, I am far from alone in griping about the sad state of router security. On June 6, 2018, security expert Bruce Schneier agreed with me, writing in the Washington Post: "Honestly, the best thing to do if you have one of the vulnerable models is to throw it away and get a new one.".

Pepwave Surf SOHO router

On their podcast, Talos employees suggested buying a router from a company that cares about security and warned that you should expect to pay a bit more for it. But, they failed to suggest any particular companies.

My recommendation for a router that is a cut above the junk most people use is the $200 Pepwave Surf SOHO router. I have used one for many years. My writeup has tons of information about the router, more than enough for anyone to decide for themselves. I am not affiliated with the vendor (Peplink) in any way. I do not make money if you buy this router.

There are some routers that are marketed based on their security, but I have not kicked the tires on any of them.

The first router to brag about security was the Turris Omnia. It has been around for a while, but is not yet available in the US.

Three Windows anti-virus companies offer supposedly secure routers: the F-Secure SENSE, the Bit Defender Box 2 and the Norton Core. All are relatively new, and it is not clear that expertise in Windows viruses carries over to routers.

Techies love pfSense but the user interface is probably too difficult for non techies. DrayTek seems to make quality routers, from what I have heard and read, but they are not cheap and are barely available in the US.

WISHING and HOPING

What the world needs now, is a "good until" date on routers - a promise from vendors to fix bugs in the router software until that date. Milk has expiration dates. Windows 7 has an expiration date. Chromebooks have them too. No one drinks milk after the expiration date and few will use Windows 7 once Microsoft stops issuing bug fixes. The same can go for routers too.

Sadly, there is no reason to expect anything to change.

by Michael Horowitz. Written June 4, 2018. Minor edits June 5th. Many updates on June 6, 2018. Updated June 7th and 8th too.