This Quality Assurance Surveillance Plan (QASP) has been developed to evaluate contractor actions while implementing the Performance Work Statement (PWS). It is designed to provide an effective surveillance method of monitoring contractor performance for each listed objective on the Performance Requirements Matrix in the call order. It also provides a systematic method to evaluate the services the contractor is required to furnish.
The contractor is responsible for management and quality control actions to meet the terms of the call order. The role of the COR/ACOR is quality assurance to ensure call order standards are achieved. The contractor shall perform all work required in a satisfactory manner in accordance with the requirements of the PWS. The COR/ACOR shall notify the CO for appropriate action if it is likely that the contractor will not achieve successful final delivery of the software code in accordance with the performance objectives and acceptable quality levels (AQLs) identified below.
The COR/ACOR will evaluate the performance objectives through surveillance as reflected below by reviews and acceptance of work products and services. As indicated, the COR/ACOR will assess progress towards the final delivered software code. Note that the performance requirements listed below are required for the final deliverable. However, the sprints and incremental delivery of code will be assessed by the Government to ensure that the contractor is on a path to successful final delivery. Deliverable or Required Services Performance Standard(s):
| Deliverable or Required Services | Performance Standard(s) | Acceptable Quality Level (AQL) | Method of Surveillance |
|---|---|---|---|
| Tested Code | Code delivered under the order must have substantial test code coverage and a clean code base. | Minimum of 90% test coverage of all relevant code. | Combination of manual review and automated testing, using agreed-upon publicly-available SaaS products. |
| Accessible | Client-side rendering must conform with section 508 standards. | 0 errors reported for 508 Standards using an automated scanner and 0 errors reported in manual testing. | http://squizlabs.github.io/HTML_CodeSniffer/ or https://github.com/pa11y/pa11y. |
| Deployed | Code must successfully build and deploy into staging environment. | Successful build with a single command. | Combination of manual review and automatic testing. |
| Documented | All dependencies (and licenses for dependencies) are listed and all major functions are documented. | All dependencies are listed and the licenses are documented. Software/source code is documented. System diagram is provided. | Combination of manual review and automatic testing |
| Available | Code must be stored in a version-controlled open-source repository. | All of the code needed to run the front end of the prototype must be available. | Technology Transformation Service (TTS) will assess code availability. |
| Research | Research activities must be conducted at regular intervals throughout the development process (not just at the beginning or end). | During the first sprint, vendor shall establish a research plan in collaboration with TTS and the Department of Housing and Urban Development (HUD). This plan must account for the use of an informed consent form, if necessary, and delivery of research-related records. In subsequent sprints, research-related records will be delivered in accordance with the aforementioned research plan. | TTS will cross-reference research-related records with other project documentation to ensure that research is properly accounted for and communicated. |
| Secure | Code must be free of medium- and high-level static and dynamic security vulnerabilities. | Clean tests from a static testing SaaS, such as Gemnasium, and from OWASP ZAP, and/or documentation explaining any false positives. | https://pages.18f.gov/before-you-ship/. |
The COR/ACOR, along with the TTS Product Lead and the product owner, will inspect all tasks required by the call order to ensure contractor compliance with the call order requirements at the conclusion of each sprint, which shall have a length of two weeks or less.
Delivery of code will occur by pull request from the contractor’s repository to the TTS repository. If inspection results are satisfactory, the pull request will be merged; otherwise, deficiencies will be noted in the pull request or through issues as described below. The COR/ACOR may find the delivery satisfactory even though further work is required, provided that the specific requirements of the sprint are met.
Unless otherwise agreed upon in writing, the contractor shall deliver research-related records to HUD in accordance with a previously agreed upon research plan. The research plan shall be delivered during the first sprint.
At the conclusion of each sprint, the COR/ACOR, along with the TTS Product Lead and the product owner, will review the completed user stories and related functionality. Incomplete or inadequate code and user stories will be noted in a mutually agreed-upon issue tracker, such as Trello or GitHub Issues, and links to each issue shared with the CO. The contractor may respond in that tracker as appropriate, addressing the accuracy and validity of the defect as well as any planned corrective action (if not already noted). The contractor team will discuss and document actions to prevent recurrence in their sprint retrospectives.
At the conclusion of the period of performance, a similar procedure will be followed to document discrepancies and to assess overall performance.
If any of the services do not conform to the call order requirements, the COR/ACOR may require the contractor to perform the services again in conformity with call order requirements. Any user stories that are not accepted must be completed in the next sprint, unless the product owner and TTS product manager agree to move it to a later sprint. The COR/ACOR shall not certify satisfactory performance for the call order until all defects have been corrected. When the defects in services cannot be corrected by re-performance, the Government may:
-
Require the contractor to take necessary action to ensure that future performance conforms to call order requirements; and,
-
Reduce the call order price to reflect the reduced value of the services performed. The COR/ACOR shall, in addition to providing documentation to the CO, maintain a complete quality assurance file. The file will contain copies of all reports, evaluations, recommendations, and any actions relating to the Government’s performance of the quality assurance function, including originals of all surveillance activity checklists. All such records will be retained for the life of the call order. The COR/ACOR shall forward these records to the CO at completion or termination of the call order.
Acceptance of services shall be based upon compliance with performance standards described in the PWS and surveillance procedures described in this QASP. Before approving/certifying any contractor invoices, the COR/ACOR will verify that all invoiced services have been performed in compliance with call order requirements. The COR/ACOR shall not certify satisfactory performance for the call order until all defects have been corrected.