Category: Penetration Testing (Page 1 of 2)

PCI Pentesting and ASV Scans

Back in the days (as in when we started PCI more than 10 years ago), when it came to testing and scans, there were probably very gray lines on it. We saw a lot of reports that came out under the guise of ‘penetration testing’ that was straight out lifted from an automated Nessus Scan or one of the free Acunetix scans available. The problem was exacerbated when these penetration testing reports were further accepted by regulatory bodies like our regulatory bank and passed by other internal/external auditors. They basically just looked at a report and if it sounded and looked technical enough then it was technical enough.

Now, PCI got the hint and released a few versions of the Penetration Testing Guidance document, the latest iteration on 2017. A big part of it talks about scoping, clarifying on qualifications and requirement 11. But one of the key features of the document is highlighted in 2.1:

This came about to stem the misconception that as long as you have completed the vulnerability scan, you can use that to pass off as a penetration testing. We still see customers going down this route, in whatever creative ways they can conjure to avoid the penetration testing exercise.

An example was this response on their external PT report stating:

“We have conducted the PT exercise based on the recently passed ASV scan report by the QSA. Since the ASV scan has passed, the penetration testing report is also considered to be passed as there are no vulnerabilities to test.”

Which is basically the philosophy that as long as the scans do not yield any high or medium vulnerabilities, i.e a passing scan, there is no longer a need to conduct any penetration testing. Their concept was simple and fairly understandable: since there are no “vulnerabilities” in the scan, there is nothing for us to ‘test’.

Of course, this was rejected by the QSA.

While there are many arguments on this matter, the simple case against this is: the scan produces potential vulnerabilities and may even miss some out that may not be reported. False negatives do exist even in commercial scanners such Qualys or Nessus (two common auto-scanners). Additionally, a passing scan does not mean no vulnerabilities, it just means there are no medium/high vulnerabilities based on a non-contextual scan to the environment. A non-contextual scan means a lot of scanners already use internal libraries in their scanning database to categorise vulnerabilities without the definition of the actual environment risk it is scanning. So to equate CVSS to the actual risk of the organisation may be too broad an assumption as some low vulnerabilities may still be able to be exploited manually. The classic example here is when we check a simple form entry password and find it is well protected and designed, technically. However, a pentester may then go out into the organisation’s forum and discover that the admin regularly upkeeps a password file in Google Drive and shares it to the entire world inadvertently. The scanner won’t discover things like that.

Therefore to simply state, just because there is a passing ASV scan, it equates to penetration testing passing, is not going to get a free pass in PCI.

Another question that many organisations come back to us, when they have their team of penetration testers doing internal testing is: Well, then how do you do a penetration test, then, if you state we cannot use the ASV report to also pass our external penetration testing?

And it would seem weird, that when I look at them and answer: wouldn’t your penetration testers be able to answer that, instead of us? So from the auditor perspective, we look at 3 things: Tools, Technique, Team.  

The tools being used are important, but not all for pentest. Just by stating you have Kali or Metasploit doesn’t necessarily mean you know how to operate it. Technique (or method) is important to document. This is key for PCI and a key difference between hackers and pentesters. A pentester would know how to document each step, inform their client and normalize and not destroy the environment. A hacker (or let’s use the more correct term cracker) would simply go in and cause as much damage as possible, depending on his/her objective. You would rarely come across crackers developing comments and detailed reports/documents to their victims and executive summaries to the Audit Committee justifying their methods, the scope of coverage and the time and date of engagement. And finally, PCI looks at the personnel (or team) conducting the exercise. They may be certified (or not), but they should at least be qualified. In this case, if the pentester has no idea how to start a pentest, then the normal assumption would be — he’s not a pentester. A chef doesn’t ask people how to start cooking. He may require an input or two to understand what he needs to cook, or how spicy the broth should be for the customer; but if the he’s asking how do we start the cooking process or what is a wok, then that should be a red flag.

So, while the coverage of penetration testing and vulnerability scanning in the entire document is not the the purpose of this article, it is keenly important to know the difference between both (penetration test vs vulnerability scan), and not use one to justify the inaction of the other. Your QSA may bounce back that vulnerability scan attempting to disguise itself as a penetration test and waste precious compliance timeline in the process.

Drop us a note at for any queries you have for PCI-DSS or ISMS and we will get back to you straight away! Stay Safe!

Penetration Testing and Vulnerability Scans

In our compliance services, oftentimes, we are tasked to assist our clients in security testing – either conducting those ourselves, or to verify previously conducted tests for compliance purposes. There are many occasions where clients decide to perform the scanning on their own, aside from the obvious option of engaging another party to do this. When we receive the test reports from our client to verify, that’s when the excitement begins.

The fundamental question we often face is, what should a penetration testing report look like? What does a vulnerability scan looks like? This age old question has been haunting PCI-DSS for years, so much so that the council decided to publish a guidance on this, found:

It’s a good read, if not fairly simplified, but it seeks valiantly to answer the question of what is a penetration testing vs vulnerability scans. This is important, because in PCI-DSS, the latter needs to be done quarterly, while the former needs to be done annually. When you multiply that by the costs and number of assets in scope, we could be looking at a decision involving tens of thousands of dollars.

In the document, section 2.1 dives into this and attempts to seek a differentiation between these two. In the basic concept of penetration testing methodology, these two activities serve specific purposes, for instance in the activities of Discovery, Enumeration, Footprinting, Exploitation, Cleanup etc, depending on which approach you take. And while there are many ways to explain the differences, to summarise:

A penetration test can be a vulnerability assessment (or scan, we will use interchangeably for the sake of this article) and beyond, while a vulnerability scan is not a penetration test.

A Penetration test can be initiated with a vulnerability assessment. The result from the vulnerability assessment will be used by the tester to penetrate or perform a more detailed assessment to circumvent controls or exploit the discovered vulnerabilities. In the process, the tester will also use manual methods to “test” the vulnerable system and likely during this process of poking around, discover more vulnerabilities or loopholes in the system that may not be detected during the initial scan. In the presentation of the findings of a penetration testing report, typically the ‘Proof of Concept’ (POC) detailing how the vulnerability was exploited will be documented.

Vulnerability assessment is the process to find out known vulnerabilities by using an (oftentimes) automated method (such as scanning software or scripts) against the targeted system. The result of the scanning will detail down the vulnerabilities, the risk exposures and action that can be taken to remediate these vulnerabilities. There is typically no manual proof of concepts that is done in the penetration test. The objective of a vulnerability assessment is to discover and report known vulnerabilities, not to exploit them.

A penetration test will normally take longer time to complete, i.e. few days, considering the manual verification or activities that need to be carried out to ‘penetrate’ the vulnerabilities. A vulnerability assessment can be completed in a shorter time frame, depending on the size of scope and software installed on the target system and it can be run on automated or scheduled basis. In our vulnerability scans, we also refine the results further by eliminating false positives, such as a patch that might not have been applied, but other secondary controls like virtual patching are in place to mitigate the risks. In either case, these are different activities, and in PCI, we need to understand what is NOT Penetration Testing.

We once received a 250 page report from our client who proudly said this was a professional work done by an outsourced security testing company offshore. Surprised as such a tome, which we assumed must have excerpts of Tolkien’s Lord of The Rings in there for good measure – we went through it. We found that it was nothing more than a raw report of the entire software inventory of the entire scope of around 50 plus assets. Meaning it listed down in excruciating detail what are the software installed in each of these systems, the licenses the OS versions etc. It was nothing more than a dump of the system’s software and nothing more. Not even the courtesy vulnerability scan. We told our bright eyed customer that we cannot accept this, and while this is a good book to have in terms of detailing the software they have, it has nothing to do with penetration testing, or vulnerability assessment. From singing praises of the offshore company, he ended up throwing them invectives that would make a pirate cringe.

We do need to be careful. We are not saying that the entire industry is filled with such charlatans peddling so called pentest services for a song and giving you a report that only provides you with the figurative emperor’s clothing for your security needs…but we must be able to differentiate what is, and what is not, security testing.

If you have further questions on security testing, drop us a note at and we can quickly assess your needs and advice you on your next options to take.

The Obfuscation of PCI Standards


When you go through the PCI-DSS standard, while in most part, the sections are clear, there are some that just annoys the heck out of me, for good reasons.

Stateful inspection and Anti-spoofing in firewalls – I know these are useful features, but it is extremely rare these days to encounter clients going for PCI-DSS that own firewalls without these capabilities inbuilt. Even the humble ScreenOS running on your tiny SGs (Juniper) are enabled by default. While this isn’t an issue, we’ve faced vexing times when our clients are sometimes asked by their QSA, to show the firewall rules that prove that Stateful inspection and anti-spoofing is turned on. We have to come in and explain to them that its already enabled by default, and they insist on us testing and showing them traffic captures. Sometimes, I just show them the manual and entitle my email “RTFM: Stateful Inspection was first introduced in 1994.” You would think that PCI would do something better than to ask this question.

AntiVirus and AntiMalware – the researchers at Imperva, a couple of years back, did a study of effectiveness of antiviruses.They collected 82 new computer viruses and ran the malware against antiviruses from some of the largest companies like Symantec, Kaspersky, McAfee. The results: initial threat detection rate was 5 percent. That’s detection. This means 95% of malware is undetected. I don’t know how strong this hypothesis is, but frankly, we have known for years antiviruses, while there are limited uses, presents to us a false sense of security. Just because the antivirus says, “ALL IS SECURED” doesn’t really mean anything. The annoyance here is not that PCI has antivirus as part of their controls, they dedicated an entire requirement to it. It’s not effective – move on!

Confusion of application testing – Requirement 6.6 states:

For public-facing web applications, address new threats and vulnerabilities on an ongoing basis and ensure these applications are protected against known attacks by either of the following methods:

a) Reviewing public-facing web applications via manual or automated application vulnerability security assessment tools or methods, at least annually and after any changes

b) Installing an automated technical solution that detects and prevents web-based attacks (for example, a web-application firewall) in front of public-facing web applications, to continually check all traffic.

Note: This assessment is not the same as the vulnerability scans performed for Requirement 11.2.

Now, we need to clarify this because this is obfuscation. Note the nice caveat they put into 11.2. Now, if you go to 11.2, you get a whole bunch of requirements for vulnerability scans, quarterly ASV etc.This is understood, right

So the above, you would surmise this: if I have a WAF (web application firewall), I do not need to do any web applications review, correct? What IS a web application review anyway? In a lot of instance, QSA will interpret it as web application testing, covering OWASP top 10. In pentest world this is called WEB APP PENTEST. This tests issues like cross site scripting, validation etc. You can find more here

A web app PT can set you back around RM10 – 25K depending on your web app and the provider. I’ve seen web applications go into the RM50 – 80K regions before for massive applications, but in general for a web application payment system, you would get that range (unless the provider is looking to rip you off, in which case I suggest you give us a buzz).

So if you have 10 – 20 Web App, that would set you back a mile, so the suggestion is to “Let’s invest in WAF”, where you pay a license and every year you don’t have that WEB APPLICATION testing headache siting on your books. In the long run, it makes more sense if you have a lot of web applications to test.

Now, here is the PCI problem.

Requirement 11.3 – Implement a methodology for penetration testing that includes the following:

blah blah blah

 – Defines application-layer penetration tests to include, at a minimum, the vulnerabilities listed in Requirement 6.5

Unfortunately the presence of 11.3 renders the earlier requirement choice useless, because now QSA interprets at despite having invested in WAF, they are still insisting on getting 11.3 passed, which requires this application layer PT!

My question to the PCI-SSC, why don’t you include this 11.3 caveat in the earlier 6.6 requirement instead of the useless 11.2 caveat which anyone knows how to read? And if my interpretation is wrong, I am going to war against some of these QSAs because they basically said, it’s nice to have WAF, but you still need to do App PT. In fact, one of them actually said: “Well, the advantage is that you are more secure.” Yes – but our client’s goal was to pass PCI. If they wanted financial modelling and investment advise from you, they would ask it, if not, just do our job and interpret the standards properly! Will someone from PCI-SSC actually clarify this because I’ve talked to some QSAs on either side of this opinion – some say WAF OR APP PT, some say, APP PT regardless of WAF.

To be safe – get your QSA to interpret this before making a decision to invest in WAF, because this is a major roadblock in a lot of cases we are in.

The Star Online Hacked


Like the entire population of Malaysia and everyone else on this planet except the few strange people from MARA (who obviously do not have children of their own, or if they do, have an extreme dull sense of what morality is) – I was keeping up with the story of the Malaysian Paedophile case. Everyone knows about it. Nur Fitri was busted and convicted as a paedophile (one who sexually abuses children – granted, he was caught with images, not actually abusing, but its just as bad), and MARA (the organisation that had given Nur Fitri the scholarship) went on the record stating that he deserves a second chance because he is a Maths Genius and an asset to the country. And that being convicted as a paedophile is like playing truant at school.


Anyway, while trying to get to the Star online, the message popup received was You’ve been hacked by the Syrian Electronic Army.


It’s actually not Star that got hacked. They attacked Gigya, a customer identity management platform that is apparently used by The Star. This is an attack that is prevalent since last year so I am not sure why Star is still having this issue. Any link to Gigya gets pointed to SEA’s images and servers. A quick look at Star’s load up and we see a whole bunch of references to Gigya.

Star, you need to remove that component from your site!

If you need help in testing your site for vulnerabilities, please contact us at


FREAK Vulnerability on Windows


As we do our penetration testing, we have to continue to get updated on some of the latest issues affecting systems out there. SSL seems to get the mother of all shares of vulnerability, with Heartbleed and then POODLE, and now, FREAK.

FREAK is found in detail at, which is basically a MiTM attack exploiting weak 512-bit keys. It affects OpenSSL, and upgrading to v1.0.2 fixes the flaw.

Basically, if you have weak cipher suites supported or SSL/TLS RSA-Export less than 512-bits, then get rid of it.

Resolution: We have always advocated to remove weak ciphers. Nobody really understood why, but now there is a vulnerability to include in our report.

If you need some assistance in vulnerability assessment, penetration testing or security audit to cover FREAK and other vulnerabilities, drop us an email at and we’ll get a team to you.

A writeup on the recent FREAK vulnerability.

Hundreds of millions of Windows PC users are vulnerable to attacks exploiting the recently uncovered “Freak” security vulnerability, which was initially believed to only threaten mobile devices and Mac computers, Microsoft Corp warned.

News of the vulnerability surfaced on Tuesday when a group of nine security experts disclosed that ubiquitous Internet encryption technology could make devices running Apple Inc’s iOS and Mac operating systems, along with Google Inc’s Android browser vulnerable to cyber attacks.

Microsoft released a security advisory on Thursday warning customers that their PCs were also vulnerable to the “Freak” vulnerability.

The weakness could allow attacks on PCs that connect with Web servers configured to use encryption technology intentionally weakened to comply with U.S. government regulations banning exports of the strongest encryption.

If hackers are successful, they could spy on communications as well as infect PCs with malicious software, the researchers who uncovered the threat said on Tuesday.

The Washington Post on Tuesday reported that and were among the sites vulnerable to these attacks, but that the government had secured them. (

Security experts said the vulnerability was relatively difficult to exploit because hackers would need to use hours of computer time to crack the encryption before launching an attack.

“I don’t think this is a terribly big issue, but only because you have to have many ducks in a row,” said Ivan Ristic, director of engineering for cybersecurity firm Qualys Inc.

That includes finding a vulnerable web server, breaking the key, finding a vulnerable PC or mobile device, then gaining access to that device.

Microsoft advised system administrators to employ a workaround to disable settings on Windows servers that allow use of the weaker encryption. It said it was investigating the threat and had not yet developed a security update that would automatically protect Windows PC users from the threat.

Apple said it had developed a software update to address the vulnerability, which would be pushed out to customers next week.

Google said it had also developed a patch, which it provided to partners that make and distribute Android devices.

“Freak” stands for Factoring RSA-EXPORT Keys.

– Source from Reuters

« Older posts

© 2024 PKF AvantEdge

Up ↑