Visa/PCI Council role?

[Shift4SMS]

What is Visa’s role, and soon to be the PCI council’s role in filed PABP/PA-DSS audit reports?

Once a QSA audits an application and files a report with the governing body, my opinion is that since the QSA program exists, the governing body should do no more than to validate the QSA’s credentials and verify the filed report was not forged. Is it defined anywhere what the governing body is responsible for as far as the filed reports are concerned? If they are double checking their QSA’s, do they not trust the program they setup?

The reason I ask is that it is taking a very long time to get filed reports through the system. Currently Visa has a 60+ day backlog for “approving” these reports and I’m not sure why this is the case. If they find anything wrong with the report or change the rules (which they have on us – we have some of the earliest PABP certified products on the list), then even more time is added. Is this the “norm” and will the PCI council, once they take control, continue down this same route?

[andrewj]

In most good certification programs, there is a disconnect between the agency that performs the evaluation, and the agency that performs the certification. This is true of PCI DSS (the QSA provides the RoC, but the Acquirer provides the certification), it is true of PCI PED (the lab performs the evaluation, and the PCI council provide the certification), and it is true of many other such certification programs, such as FIPS 140-2 and Common Criteria.

This disconnect is essential to ensure that there is partity between the tests being performed by the different evaluation agencies, and that all aspects have been thoroughly tested. It is the job of the certification agency to review the report and ensure that the testing is complete, and ask any questions where areas may be unclear, or they feel further testing is necessary.

I can assure you that when you have knowledgeable people in the certification body, this process works very well.

[Shift4SMS]

But how can the certification agency make a determination as to whether or not an auditors opinion is valid without understanding the technology or at least looking at it?

Ok, I'll put on the "it all is good" hat and assume all the checks and balances are working properly -- are there any plans to eliminate the backlog? This problem is going to get a lot worse as the various deadlines approach requiring all payment applications to be audited and certified. The delay was about 30 days last year, sixty days this year, 120 next?

[andrewj]

But how can the certification agency make a determination as to whether or not an auditors opinion is valid without understanding the technology or at least looking at it?

By understanding the requirements, and understanding how systems are commonly compromised in the field, and ensuring that the testing has taken into account such possible attack vectors. For instance, the report should be clear about how the testing confirmed that card data is not stored by the software - the certification body may find this ambigous, or incomplete, and ask for either clarification, or additional testing.

are there any plans to eliminate the backlog?

I'm sure there are such plans, but as with all things, I am also sure that a deadline will cause some time pressure that may reveal itself in any of a number of ways. I can understand your frustration, but I am not sure what can be done at this stage. If you are a participating organisation with the PCI SSC, I would suggest you persue them through official channels to obtain some understanding of how the process will work when they take over.

[Shift4SMS]

In most good certification programs, there is a disconnect between the agency that performs the evaluation, and the agency that performs the certification.I was thinking about this more like a CPA's. When a CPA audits your books, there isn't another agency that certifies the audit. If the CPA is found to be "certifying" fraudulent books or not following whatever CPA laws and ethics, they can lose their license, but for the most part there is no auditor of the auditor.

Can someone help me understand the money flow? Obviously money makes the world go around. I pay for a QSA or QPASP to audit me. Who pays the controlling agency that scrutinizes the report my auditor submits? The reason I ask is because of my concern about certification time frames. I don't see the time frames getting shorter without more manpower at the controlling agency and manpower requires money.

[sm1978]

I was thinking about this more like a CPA's. When a CPA audits your books, there isn't another agency that certifies the audit. If the CPA is found to be "certifying" fraudulent books or not following whatever CPA laws and ethics, they can lose their license, but for the most part there is no auditor of the auditor.

This is true for accounting agencies, remember though these are assessment not audits. The bar to do an audit is much higher, an assessment not so much, let alone the quality of work that may be coming out of these assessors. Trust me, I have a bad one right now, the quality is hit/miss. I think any good body managing these assessors have a legitimate QA process in place to ensure clarifications are asked for when there is ambiguity or completely missing the requirements.

[lyalc]

Don't forget, there are processes to include the vendors confirmation of report accuracy in PABP, QSA feedback forms in PCI, and the option to directly contact Visa/PCI SSC also exists.

All these sound a lot like the CPA 'checks and balances'. You have a voice - be heard, and if your complaint is reasonable (and evidence supports it), I reckon steps will be taken.

lyalc

[Shift4SMS]

This is true for accounting agencies, remember though these are assessment not audits. The bar to do an audit is much higher, an assessment not so much, let alone the quality of work that may be coming out of these assessors. Trust me, I have a bad one right now, the quality is hit/miss. I think any good body managing these assessors have a legitimate QA process in place to ensure clarifications are asked for when there is ambiguity or completely missing the requirements.Hmmm, maybe it's my misunderstanding of the process then. I've always been told and always assumed a PABP audit of the application was being performed. In reading the PABP requirements, I do now see that it states an assessment must be performed. So I see your point.

In that case and considering your comment about the quality inconsistencies between assessments, now I question the whole QPASC program. I assumed that all the QPASC's on the Visa list would provide consistent assessments.
What exactly am I paying for and is there another list of quality QPASC's vs. the iffy ones?

For me, the money I pay a QPASC is not the issue (although, it could be if the price tag is totally outrageous). The bigger issue for me is time. Each round takes 30-60 days (file report, wait for review and results, address issues and re-file; this assumes no software reengineering time). If this bounces back and forth 2, 3, 4 times, we're talking roughly 3-9 months to get through this process. This is intolerable for the average software vendor (this time is added to the normal software development cycle since a PABP assessment cannot be made on beta code). Now if I can find a QPASC that can all but guarantee a single filing, then this whole program might be tolerable.

[sm1978]

Now if I can find a QPASC that can all but guarantee a single filing, then this whole program might be tolerable.

My company is in the same shoes as you it sounds like, these QSAs are a mixed bag. I'm sure it's the same for the QSAs doing the PABP. I wish quality was universally across the board that just isn't going to happen, some of these companies will try to get in and out ASAP, some will be thorough and take more time, some just skate through to get the minimum done. You may want to get some referrals and see if what you're getting is worth what you're paying, we're in that process now. What is your QPASC telling you about these delays?

[lyalc]

It sounds like a combination of 2 basic issues here
1. Vendors trying to the the least amount of work/cheapest compliant solution, and thus the incrementally trying to get an acceptable report
2. Assessors not being firm enough on the intent of PABP and or PCI before sending off the assessment report.

Possibly a third issue also - assessors who have no experience in installing/managing systems or security, so not appreciating how a product will look and feel, out there in the real world.

I know - these can be hard pressures to resist all the time.

lyalc

[Shift4SMS]

It sounds like a combination of 2 basic issues here
1. Vendors trying to the the least amount of work/cheapest compliant solution, and thus the incrementally trying to get an acceptable report
2. Assessors not being firm enough on the intent of PABP and or PCI before sending off the assessment report.

Possibly a third issue also - assessors who have no experience in installing/managing systems or security, so not appreciating how a product will look and feel, out there in the real world.
I totally disagree here. This may be true in some cases but not here. In this particular example that inspired this thread, the solution being assessed is so "non-standard" that it does not fit into the shoes that Visa deems all POS applications should use. I have other opinions on this subject as well but I'll hold off because it would make me come off as anti-PCI and I'm really not -- I just think there are some big problems that need addressing.

By any chance are you a QPASC? If so, PM me your info. I would be VERY interested in comparing the assessments and the assessment process results.