Much like the way that Stephen Colbert uses Truthiness, where it is not the actual truth but “the quality of preferring concepts or facts one wishes to be true, rather than concepts of facts known to be true”. Complianciness is not actually compliance, but that you want to believe that you are compliant rather than actually being compliant.

This word evolved from a conversation with a number of highly regarded professionals (Mike Smith, Dan Philpott and Graydon McKee), with shout-outs to Chris Hoff and Anton Chuvakin for a twitter exchange around measurements (I believe this post is worth 286,497 Chuvakins).

And here (I believe) is one of its best examples:

Recently, Bob Carr (CEO of Heartland Payment Systems) gave a podcast interview to Bill Brenner at CSO Online. During this interview, Carr basically threw his Qualified Security Assessors (QSA) under the bus. He had equated what the QSA is responsible for doing (measuring Heartland against the PCI controls) for actual secure operations. There are two sides to this story I am sure, however I like to think of PCI-DSS as a “tech heavy” control set. To me this means that the controls are very focused on securing the technology but that it could still be maintained or operated poorly. That also means, that the QSA is focused on a very specific set of controls, not on Information System Risk Management (although it does mandate an annual risk assessment).

Carr asserts that the QSA assessments were not helpful for the years that he had to pay for it. I would argue that they were helpful because it showed Visa and Mastercard that Heartland had implemented a minimum set of controls to protect some information for Visa and Mastercard.

Where's the complianciness? Heartland Payment Systems – based on my research of the situation Heartland may have been PCI compliant at the point in time that they were assessed. It could be that security was a little more lax when the assessors were not inbound to conduct testing. It also might have been that a very elaborate show was put on for the assessors and they were not actually compliant, but perhaps practicing complianciness.

Question from the Audience: I thought you only talked about FISMA, where is the complianciness there?

A: Thank you random voice in my head.

It is rampant. FISMA / OMB / NIST Guidance all used come back to one thing, Information System Risk Management. The process of identifying risks to your mission or system and then applying a specific set of policies and controls to mitigate those risks. What has evolved from this, and I only have anecdotal evidence / hearsay to back this up, is that integrators, developers and operators are being told to simply follow NIST guidance. When that is entirely not possible without making some decisions by the customer.

NIST guidance is there to provide options and support. It can lay out things to consider and questions you should be asking yourself. The policies, procedures and processes that are carried out on an information system must be clearly defined at the beginning of the design process by being built into the design requirements.

I feel like now is the best time to mention these ideas because many organization in the government are going to be reviewing NIST 800-53 Rev 3, and having to make some decisions about what their policies are going to be for the next couple of years. My response is Choose Compliance not Complianciness.

Telling a vendor to go look at NIST and build it, will only get you what the vendor is willing to provide by shoehorning the solution into the controls. I'll elaborate. Vendor will read the controls. Vendor will either write the SSP or provide input to the Self Assessment. Those result will be their interpretation of the least effort required by that control. As the federal customer, you will believe that they have implemented that control to the strength that you believe it should be implemented. You have therefore practiced complianciness, at least until the auditor or assessor comes in and tells you what is actually happening (you hope). Because the auditor could in fact perceive the controls to be something entirely different altogether.

If you are part of a government agency, you want to provide a policy that can be based on 800-53 but you need to answer the questions it asks.

And that's the word.

I don't mean to be an alarmist or whatever. But that's how newspapers get sold and stations get ratings. What is this mystical issue? The answer is Control Implementation Prioritization.

Way back in February 2009 we were greeted with the Consensus Audit Guidelines (CAG). I personally do not care for CAG. Some people will get their controls implemented faster, better, cheaper. At a minimum, the guidelines are misleading since it had little to do with actual auditing or system security testing.

At the beginning of May, they revised CAG into 20 Critical Security Controls (CSC). Well at least now, there is some truth in the title. They are sold to us as controls that every system should implement. Well ... thanks. Let's take a quick look then.

Ahhh. Ok. So where is the part about laying down a strategy or developing an initial policy that needs to be followed. Its not there. Where is the part about strength and cost of the control implementation as measured against the risk. I couldn't find that either.

Apparently, that doesn't matter anymore. It is clear from the beginning that the focus of CSC is not about system-specific risk analysis anymore. So that is that, but then in Appendix D of the 800-53 Rev 3, Final Public Draft. What do my eyes find, CONTROL PRIORITIZATION. On a scale of 1 to 3 and a 0 for unspecified.

Now for the meat - why is this bad. Its bad because management types will focus on the number 1. "I have to do these controls first, because NIST told me so". Or "I have money for the top 20 then I will deal with the rest".

It has been proven time and time again security comes from determining risk and implementing controls comensurate with that risk. Then reassessing that risk and control effectiveness over time using adequate metrics. Plan for the worst with contingency and incident handling plans. Et cetera.

Implementation of the CSC will not make you safer, it will make the vendor richer. A total soup-to-nuts program is still the only way from my opinion. This would include selecting controls that you deem necessary.

... I am just extremely busy. Please don't remove me from your RSS feed just yet.

In the interim, you can check me out at:
http://blog.marcusjcarey.com/2009/04/that-security-show-news-segment-concept.html

http://blog.marcusjcarey.com/2009/04/that-security-show-sampler.html

Random thoughts:

The ICE Act isn't going to make us more secure;
Mowing the lawn sucks;
I really like teaching at the Potomac Forum;
I am working on a concept/paper that I am calling Big Risk and;
I am also working a compliance framework to help manage security and compliance testing.

That is all.

I was recently using the twitter machine when someone asked me how I would develop requirements and the subsequent test cases for embedded devices. Beyond the fact that I needed more than 140 characters to answer, I found the question simultaneously amusing and befuddling. So this post is the result of that initial query.

We know that embedded devices will not have all the controls that even something like Windows is capable of meeting. Windows has a difficult time trying to meet a FIPS 199 categorization of moderate. Therefore, these devices put us (me) into a quandary. The podcast Pauldotcom.com routinely talks about pen testing exploits that involved using an embedded device as a launch point for more sinister attacks. But the devices will never have the security controls that full blown operating systems and applications are capable of implementing.

We also know that these types of devices have striped down versions of things we already know and love, like ... TCP stacks and ... file systems. But the tools that assessors or testers use with servers, web sites and routers do not work (at all) or unreliably (at best).

So here we have devices that are in the system boundary and processing data. Prevalent security researchers have already demonstrated the issues with them.

Obviously, they need to be tested; they need controls and protections. But how to test while collecting this mysterious assurance evidence. The answer is the dreaded manual test case. Sitting down with your refrigerator or microwave with your requirements (let's say its an agency tailored 800-53). Sit with the vendor or poor sap who has been tagged to "be in charge" to walk through the system with you as you develop the test steps. You are not retrieving the results or collecting evidence yet. This is merely to work out a repeatable process by which others can use to re-test later.

You now want to ask me: "what about requirements that I can't develop test steps?" So a control is not in place no matter what. This is still a requirement. It just means that you don't have to test for it because it has already failed. But you will need to leave a spot in the Security Assessment procedures that says "I interviewed and the vendor/system could not provide evidence that this control could be satisfied." OR "Review of manuals and system documentation revealed that the system does not implement the control" Fail. It does not mean that it is Not Applicable, because it is still a requirement.

What about gathering proof that the control is actually in place? This is what I think the real question is; the answer is that it depends. If you are going through a terminal, then you can capture the session to a text file. If it can be remote controlled through something like VNC or RDP, then you could take a screen movie. I found this software today which they claim you can embed into Word or PDF.

But then there are those that there is no remote screen or remote terminal. All we have is a generic interface on the device itself. Well I don't know what to tell you there except camera. Oh yes. The dreaded video camera on a tripod. You will need waivers and exemptions and all kinds of paperwork. But it is really the only way to capture the test procedure if that's the level of assurance required. That's why I left it for last, because it is most unpleasant. This would also fall in the category of "evidence available upon request".

Hopefully, a detailed procedure is all you need. Here is a sample of what I would envision a test of account lockout (AC-7) to look like (but it is lacking my usual pretty formatting):

Step 1: Log in using normal interface with a valid user account and password combination.
Expected Result: Log in successful

Step 2: Log out and attempt to log in using a valid user account and invalid password combination.
Expected Result: Log in unsuccessful

Step 3: Re-attempt Step 2 until .
Expected Result: Log in unsuccessful

Step 4: Re-attempt Step 1
Expected Result: Log in unsuccessful

Step 5: Wait for minutes (Only if not unlimited) and repeat Step 1
Expected Result: Log in successful

My typical reaction is to stop a procedure once something has failed. Or to have dependencies in the test steps to limit the number of procedures I have to manage.

So I don't know if I answered the original question, I feel better for putting at least something out there.

I gave a Fire Talk at ShmooCon. I had hoped to convey that the way Federal agencies have been conducting their security assessments, has been flawed (at best) or wrong. I was asked a simple question "What should we do to fix it?" I gave a stock answer like document test cases better and spend more time and money on the assessment in general. It was a blow off but it was the best I could come up with while simultaneously being scared sh**less.

Then, I also was fortunate enough to be an instructor with the Potomac Forum at their Certification Accreditation Workshop with Graydon McKee and Dan Phillpott. It was truly awesome and glorious two days, but I digress. I was in the middle of a diatribe about how to assess a Federal system under the current NIST guidance and FISMA. I got to a part where I started talking about running a penetration test on the system before the accreditation/authorization to operate. Then another question "Do we NEED to run a pen test?" To which I responded ... "it depends".

Given the money and the time I would have any system I worked on penetration tested. I spent a few minutes trying to find what I mean when I say "pen test". I found it on this site, but I have it for you here:

Penetration test - A test of a network's vulnerabilities by having an authorized individual actually attempt to break into the network. The tester may undertake several methods, workarounds and "hacks" to gain entry, often initially getting through to one seemingly harmless section, and from there, attacking more sensitive areas of the network.

Who wouldn't want that? I would also add to this definition that the vulnerability is actually exploited and that evidence of the exploit it captured. Because then you have actually tested something. Running a tool and saying something like "conditions are favorable for a successful exploitation of" ... blah blah blah, is not a penetration test. That is a vulnerability assessment.

The reality is that most systems should be having vulnerability assessments done monthly, if not more frequently. Its not policy, but that's my opinion. Penetration test annually, or after substantial changes to the architecture.

Excuse 1: It Expensive.
Response: So is loosing your data.

According to a recent study (that I am currently unable to find - if you have a link then please comment), it could cost something like $200 per customer to restore their good standing. A decent test by a rock star tester is pricey. If we use this $200 number. Well - how many customers are using the system? Times Bad PR + Sleepless nights + Incident Response Services = a crap load more than the Pen Test.

Excuse 2: They could break our shhhh ... stuff
Response: We'll schedule downtime.

Most pen testers love pen testing. They also like money. Most will probably work with you to sacrifice a Saturday or Sunday evening. For one or both of these reasons. The other reality of this statement is that the person generating this excuse could be afraid of what is found. Ignorance is not bliss and obscurity is not security, the attacker will find weaknesses in the system. Get over the pride and let's just fix it.

Excuse 3: Our Coders / Developers are awesome
Response: Awesome people still make mistakes

I don't presume that most of the people who would read this trust a Bank carte blanch to handle your finances. You probably reconcile your check book, make sure that your online banking bills do, in fact, get paid, etc. Humans are not error-free neither is your code or system design. New vulnerabilities are found every day in software that we all use.

Excuse 4: We don't have time before the system needs to be live
Response: Get one after the system is live

The attackers will be working on your system from the word go. You will be required to defend it. Return to reasoning for excuse 2.

Excuse 5: Nobody wants our data
Response: You don't have a competitor?

Competitor is a wide range of possibilities. The Federal government has not just competition but real enemies. It could be another nation, terrorist or garden variety kook. If you aren't just putting information on the Internet, then clearly it requires protecting. Also, an attacker has time. Conceivably, they are motivated and they want what you have. They will spend hours, days or years working on your system and basically you need a way to outlast them.

Boss: Ok, I'm in. What's next?
You: Ahhh, yeah. I'll send you an email in the morning.

When in fact your response should have been: We are going to test everything. The guys we are bringing in can:

• Attempt compromise from the Internet;
• Attempt compromise from the inside (insider threat and/or accidents);
  
• Social Engineer our employees and service providers (that's right I said it);
• War-dial, war-drive, war-walk, war-unicycle through and around our facilities to identify unknown network entry points;
• Leave USB thumb drives in the parking lot or FedEx DVDs or CDs to insiders;
  
• (Please Comment to add more)

You may not need to do all of this each time, but it is my opinion that every organization should be going through most of these exercises on a regular basis. While they don't all fit the definition of a penetration test, these services can generally be provided by the same firm. You won't find anything in FISMA or NIST or OMB that says: "Thou shalt get a pen test", but the 800-42 says its a good idea.

So go get a pen test .. now.