Software Supply Chain Security

The Executive Order (EO) on Improving the Nation’s Cybersecurity (14028) directs NIST to publish a variety of guidance that would enhance software supply chain security. Among other deliverables, NIST published these documents for public comment:

NIST earlier produced other guidance to improve software security:

Recommended Minimum Standards for Vendor or Developer Verification (Testing) of Software Under Executive Order (EO) 14028

Introduction

Executive Order (EO) 14028 on Improving the Nation’s Cybersecurity, May 12, 2021, directs the National Institute of Standards and Technology (NIST) to publish guidelines on vendors’ source code testing.

“Section 4(r) Within 60 days of the date of this order, the Secretary of Commerce acting through the Director of NIST, in consultation with the Secretary of Defense acting through the Director of the NSA, shall publish guidelines recommending minimum standards for vendors’ testing of their software source code, including identifying recommended types of manual or automated testing (such as code review tools, static and dynamic analysis, software composition tools, and penetration testing).”

NIST solicited position papers from the community, hosted a virtual workshop to gather input, and consulted with the National Security Agency (NSA) to develop the recommended minimum standards as well as supplementary material to put the standards in the context of a robust testing program which, in turn, is part of a robust development process. 

NIST has developed a document that recommends minimum standards for vendor or developer verification of software. These guidelines are summarized on this webpage. See FAQ #3 and FAQ #4 for an explanation of why NIST added the terminology developers and verification.

Note that NIST will be developing guidance on software testing tools and attestations under Part 4(e) of the EO. See FAQ #1

This webpage provides background information and context for minimum standards for software verification. It then defines eleven tasks and techniques which comprise the recommended software verification minimums. The twelfth task, fixing critical bugs, is included for completeness.

Recommended Minimum Standards for Vendor or Developer Verification (Testing) of Software Under Executive Order (EO) 14028 - PDF Version

Recommended Minimum Standards for Vendor or Developer Verification (Testing) of Software Under Executive Order (EO) 14028 - Background & Approach

To ensure that software is sufficiently safe and secure, the software must be designed, built, delivered, and maintained in accordance with best practices. Frequent and thorough testing by developers as early as possible in the software development life cycle (SDLC) is one critical practice. At its highest conceptual level, verification is a discipline employed to increase software security. Verification encompasses many static and active assurance techniques, tools, and related processes to identify and remediate security defects while continuously improving the methodology and supporting processes. They must be employed alongside other methods to achieve a high level of software security.

This webpage summarizes the minimum standards recommended for verification by software vendors or developers. No single verification standard can encompass all types of software testing, be specific and prescriptive, and present efficient and effective testing. Thus, this document recommends high-level guidelines for software producers to create their own prescriptive processes.

These guidelines expand on NIST’s Secure Software Development Framework (SSDF) practices. See especially Produce Well-Secured Software (PW) Practice 7, Review and/or Analyze Human-Readable Code to Identify Vulnerabilities and Verify Compliance with Security Requirements, and PW Practice 8, Test Executable Code to Identify Vulnerabilities and Verify Compliance with Security Requirements. 

Recommended Minimum Standard for Vendor or Developer Verification of Code

The following are recommended minimums for verification of code by developers. Some of the minimums are the precursors to effective testing and some are the logical outcomes of the testing. Each of the techniques is described in the accompanying document.

Technique Class

Technique

Description & Reference to Recommended Minimums Document

Threat modeling

Threat modeling helps identify key or potentially overlooked testing targets.

Section 2.1. Threat modeling methods create an abstraction of the system, profiles of potential attackers and their goals and methods, and a catalog of potential threats. Threat modeling can identify design-level security issues and help focus verification.

Automated testing

As testing is automated, it can be repeated often, for instance upon every commit or before an issue is retired.

Section 2.2. Automated testing can run tests consistently, check results accurately, and minimize the need for human effort and expertise. Automated testing can be integrated into the existing workflow or issue tracking system.

Code-based (static) analysis

Use a code scanner to look for top bugs.

Section 2.3. Static analysis tools can check code for many kinds of vulnerabilities and for compliance with the organization’s coding standards. For multi-threaded or parallel processing software, use a scanner capable of detecting race conditions.

Review for hardcoded secrets. 

Section 2.4. Heuristic tools can be somewhat effective checking for hardcoded passwords and private encryption keys since functions or services taking these as parameters have specific interfaces.

Dynamic analysis (i.e., run the program on test cases)

Run with built-in checks and protections.

Section 2.5. Programming languages, both compiled and interpreted, provide many built-in checks and protections.

Create “black box” test cases.

Section 2.6. “Black box” tests can address functional specifications or requirements, negative tests (invalid inputs and testing what the software should not do), denial of service and overload attempts, input boundary analysis, and input combinations.

Create code-based structural test cases.

Section 2.7. Code-based, or structural, test cases are based on the implementation, that is, the specifics of the code. Code-based test cases may also come from coverage metrics.

Use test cases created to catch previous bugs.

Section 2.8. Test cases which have been created to specifically show the presence (and later, the absence) of a bug can be used to identify issues in the absence of more general “first principles” assurance approaches for detecting bugs.

Run a fuzzer.

Section 2.9. Fuzzers can try an immense number of inputs with minimal human supervision. The tools can be programmed with inputs that often reveal bugs, such as very long or empty inputs and special characters. 

Main Menu