Embedded Compliance

I was recently using the twitter machine when someone asked me how I would develop requirements and the subsequent test cases for embedded devices. Beyond the fact that I needed more than 140 characters to answer, I found the question simultaneously amusing and befuddling. So this post is the result of that initial query.

We know that embedded devices will not have all the controls that even something like Windows is capable of meeting. Windows has a difficult time trying to meet a FIPS 199 categorization of moderate. Therefore, these devices put us (me) into a quandary. The podcast Pauldotcom.com routinely talks about pen testing exploits that involved using an embedded device as a launch point for more sinister attacks. But the devices will never have the security controls that full blown operating systems and applications are capable of implementing.

We also know that these types of devices have striped down versions of things we already know and love, like ... TCP stacks and ... file systems. But the tools that assessors or testers use with servers, web sites and routers do not work (at all) or unreliably (at best).

So here we have devices that are in the system boundary and processing data. Prevalent security researchers have already demonstrated the issues with them.

Obviously, they need to be tested; they need controls and protections. But how to test while collecting this mysterious assurance evidence. The answer is the dreaded manual test case. Sitting down with your refrigerator or microwave with your requirements (let's say its an agency tailored 800-53). Sit with the vendor or poor sap who has been tagged to "be in charge" to walk through the system with you as you develop the test steps. You are not retrieving the results or collecting evidence yet. This is merely to work out a repeatable process by which others can use to re-test later.

You now want to ask me: "what about requirements that I can't develop test steps?" So a control is not in place no matter what. This is still a requirement. It just means that you don't have to test for it because it has already failed. But you will need to leave a spot in the Security Assessment procedures that says "I interviewed and the vendor/system could not provide evidence that this control could be satisfied." OR "Review of manuals and system documentation revealed that the system does not implement the control" Fail. It does not mean that it is Not Applicable, because it is still a requirement.

What about gathering proof that the control is actually in place? This is what I think the real question is; the answer is that it depends. If you are going through a terminal, then you can capture the session to a text file. If it can be remote controlled through something like VNC or RDP, then you could take a screen movie. I found this software today which they claim you can embed into Word or PDF.

But then there are those that there is no remote screen or remote terminal. All we have is a generic interface on the device itself. Well I don't know what to tell you there except camera. Oh yes. The dreaded video camera on a tripod. You will need waivers and exemptions and all kinds of paperwork. But it is really the only way to capture the test procedure if that's the level of assurance required. That's why I left it for last, because it is most unpleasant. This would also fall in the category of "evidence available upon request".

Hopefully, a detailed procedure is all you need. Here is a sample of what I would envision a test of account lockout (AC-7) to look like (but it is lacking my usual pretty formatting):

Step 1: Log in using normal interface with a valid user account and password combination.
Expected Result: Log in successful

Step 2: Log out and attempt to log in using a valid user account and invalid password combination.
Expected Result: Log in unsuccessful

Step 3: Re-attempt Step 2 until .
Expected Result: Log in unsuccessful

Step 4: Re-attempt Step 1
Expected Result: Log in unsuccessful

Step 5: Wait for minutes (Only if not unlimited) and repeat Step 1
Expected Result: Log in successful

My typical reaction is to stop a procedure once something has failed. Or to have dependencies in the test steps to limit the number of procedures I have to manage.

So I don't know if I answered the original question, I feel better for putting at least something out there.

To Pen Test or not to Pen Test .. that is the question.

I gave a Fire Talk at ShmooCon. I had hoped to convey that the way Federal agencies have been conducting their security assessments, has been flawed (at best) or wrong. I was asked a simple question "What should we do to fix it?" I gave a stock answer like document test cases better and spend more time and money on the assessment in general. It was a blow off but it was the best I could come up with while simultaneously being scared sh**less.

Then, I also was fortunate enough to be an instructor with the Potomac Forum at their Certification Accreditation Workshop with Graydon McKee and Dan Phillpott. It was truly awesome and glorious two days, but I digress. I was in the middle of a diatribe about how to assess a Federal system under the current NIST guidance and FISMA. I got to a part where I started talking about running a penetration test on the system before the accreditation/authorization to operate. Then another question "Do we NEED to run a pen test?" To which I responded ... "it depends".

Given the money and the time I would have any system I worked on penetration tested. I spent a few minutes trying to find what I mean when I say "pen test". I found it on this site, but I have it for you here:

Penetration test - A test of a network's vulnerabilities by having an authorized individual actually attempt to break into the network. The tester may undertake several methods, workarounds and "hacks" to gain entry, often initially getting through to one seemingly harmless section, and from there, attacking more sensitive areas of the network.

Who wouldn't want that? I would also add to this definition that the vulnerability is actually exploited and that evidence of the exploit it captured. Because then you have actually tested something. Running a tool and saying something like "conditions are favorable for a successful exploitation of" ... blah blah blah, is not a penetration test. That is a vulnerability assessment.

The reality is that most systems should be having vulnerability assessments done monthly, if not more frequently. Its not policy, but that's my opinion. Penetration test annually, or after substantial changes to the architecture.

Excuse 1: It Expensive.
Response: So is loosing your data.

According to a recent study (that I am currently unable to find - if you have a link then please comment), it could cost something like $200 per customer to restore their good standing. A decent test by a rock star tester is pricey. If we use this $200 number. Well - how many customers are using the system? Times Bad PR + Sleepless nights + Incident Response Services = a crap load more than the Pen Test.

Excuse 2: They could break our shhhh ... stuff
Response: We'll schedule downtime.

Most pen testers love pen testing. They also like money. Most will probably work with you to sacrifice a Saturday or Sunday evening. For one or both of these reasons. The other reality of this statement is that the person generating this excuse could be afraid of what is found. Ignorance is not bliss and obscurity is not security, the attacker will find weaknesses in the system. Get over the pride and let's just fix it.

Excuse 3: Our Coders / Developers are awesome
Response: Awesome people still make mistakes

I don't presume that most of the people who would read this trust a Bank carte blanch to handle your finances. You probably reconcile your check book, make sure that your online banking bills do, in fact, get paid, etc. Humans are not error-free neither is your code or system design. New vulnerabilities are found every day in software that we all use.

Excuse 4: We don't have time before the system needs to be live
Response: Get one after the system is live

The attackers will be working on your system from the word go. You will be required to defend it. Return to reasoning for excuse 2.

Excuse 5: Nobody wants our data
Response: You don't have a competitor?

Competitor is a wide range of possibilities. The Federal government has not just competition but real enemies. It could be another nation, terrorist or garden variety kook. If you aren't just putting information on the Internet, then clearly it requires protecting. Also, an attacker has time. Conceivably, they are motivated and they want what you have. They will spend hours, days or years working on your system and basically you need a way to outlast them.

Boss: Ok, I'm in. What's next?
You: Ahhh, yeah. I'll send you an email in the morning.

When in fact your response should have been: We are going to test everything. The guys we are bringing in can:

• Attempt compromise from the Internet;
• Attempt compromise from the inside (insider threat and/or accidents);
  
• Social Engineer our employees and service providers (that's right I said it);
• War-dial, war-drive, war-walk, war-unicycle through and around our facilities to identify unknown network entry points;
• Leave USB thumb drives in the parking lot or FedEx DVDs or CDs to insiders;
  
• (Please Comment to add more)

You may not need to do all of this each time, but it is my opinion that every organization should be going through most of these exercises on a regular basis. While they don't all fit the definition of a penetration test, these services can generally be provided by the same firm. You won't find anything in FISMA or NIST or OMB that says: "Thou shalt get a pen test", but the 800-42 says its a good idea.

So go get a pen test .. now.