An official website of the United States government
Here's how you know
A .mil website belongs to an official U.S. Department of Defense organization in the United States.
A lock (lock ) or https:// means you’ve safely connected to the .mil website. Share sensitive information only on official, secure websites.

The Cyber Defense Review

Applied Research in Support of Cyberspace Operations: Difficult, but Critical

By LTC Stoney Trent, Dr. Robert Hoffman, Dr. Scott Lathrop | May 02, 2016

Cyberspace operations is not cyber security 

The importance of cyber security is manifest across our traditional enterprise information technology systems. Our human and military affairs are network enabled, and therefore potentially vulnerable. The reach of cyber-physical systems is also expanding rapidly, going beyond infrastructure (e.g., the utilities, water resource management, and food supply protection) and entering everything from automotive technology, to household and healthcare networks. A major effort is underway to develop the next generation of cyber security experts through the establishment of ‘academic centers of excellence’ and cyber security programs at a great many US colleges and universities. This matter is particularly acute because the current workforce must possess a great deal of knowledge and a high degree of skill.

Cyber security as a work domain and commercial sector has been maturing rapidly over the past 20 years. Many of the skills and tools required for cyber security have application for cyberspace operations. However, the Department of Defense (DoD) cannot rely on cyber security solutions to satisfy its full range of operational requirements. Cyber security is primarily concerned with keeping computer networks operational and trustworthy. Maintaining network confidentiality, integrity and availability are the guiding principles for security policies and behaviors. Small groups of experts continue to operate and protect well-defined networks and data with their tailored cyber security tools and tradecraft.

Cyberspace operations, on the other hand, are synchronized military activities to identify, degrade and/or deceive threat actors in cyberspace. Joint Publication 3-12 defines cyberspace operations as “the employment of cyberspace capabilities where the primary purpose is to achieve military objectives or effects in or through cyberspace.” In 2009, the DoD established U.S. Cyber Command (USCYBERCOM) to centralize command of joint cyberspace operations, and in 2013 initiated a build-out of a robust Cyber Mission Force (CMF) that is manned, trained, and equipped by the Service Departments.

Cyberspace operations are inherently dynamic due to changing technology and tactics of malicious actors. Recent increases in the number and scale of cyber incidents have illustrated the need for improved coordination across the CMF as well as improved feedback and accelerated technology transition between operational research, and development communities.


Effective, operationally focused research is difficult

Cyberspace operations concepts are still in the formative stage. This spans everything from organizational design within the Service Departments, to joint force coordination, to operational design. Communication and collaboration methods are evolving. The increasing scale and tempo of cyberspace operations matches new decision support tools developed to assist in network mapping, netflow monitoring, and malware detection. Thus, there is a need for research to explore work methods and objectively evaluate new technologies. There is a need for performance evaluation, and metrics that are specific to the emerging technologies. What is a good baseline for performance in cyberspace operations? Information Technology provides necessary but insufficient benchmarks for metrics such as ‘detect in time to prevent’. Our cyber community requires effective measures that evaluate the cognitive workload and operational impact at the ‘systems of systems’ level.

Enterprise-wide expertise is critical, but ephemeral. Until the DoD adopts different personnel policies, the CMF will be staffed with predominantly novice military service members supervised by a few experienced personnel. To complicate matters, many of the most proficient practitioners find lucrative employment in the private sector. Training for cyberspace work requires experiential learning at realistic cyber work activities that have high “cognitive fidelity”. That is, the training experience must place the same demands on attention, reasoning, decision-making, and interaction as actual practice. To be realistic, cyber work simulations must provide abundant opportunities for trainees to engage in network analysis, vulnerability analysis, malware detection, and other cyber activities, on realistic (although often virtual) networks. However, it is not possible to develop scenarios and simulations without establishing (and maintaining) a repository of expert knowledge, and a library of lessons-learned.

Thus, operationally focused applied research is required for converging reasons. Research focused on cyberspace operations must be informed by a knowledge of how experts, journeymen, and apprentices work together to accomplish their tasks. This mandates cognitive task analyses and knowledge elicitation. The cyber professional’s workplace technologies (software systems, visualization systems) shape their strategies and knowledge. This mandates evaluations of the strengths and limitations of automation as well as human-system performance in terms of robust metrics.

Applied research not only dissolves the traditional distinction between basic and applied science, but it necessarily blurs the distinction between research and operations.[1] One cannot easily take actual military cyberspace work out of context, situate it in a laboratory, and study it under controlled circumstances. To achieve a science-based and yet genuine understanding of cyber work one must to some degree let the ‘fog of war’ intrude into the research setting.

In addition to the integration of research and operational communities in experimental applied research settings, the involvement of the development community who might ‘productize’ or engineer the final integrated product is critical to facilitating timely transition and sustainment of technological breakthroughs. The software developers who will eventually integrate the research into a system can provide the research community key insights from an engineering perspective that will enable more rapid technology transition. Because of their day-to-day hands-on involvement with various tools, the developer community may also provide operational use cases that the research or operational community may not always perceive.

Integrated research and training enables a decision support system that is sufficient for training and should also be sufficient for conducting the actual work, and vice versa.[2] Exploratory and evaluative work analysis conducted in the same context as cyberspace training, assures cognitive fidelity. Experience shows that research and training in a ‘virtual distance’ mode does not entail the complexities, practical issues, and networking issues involved in ‘actual distance’ mode. Simply saying, “Room 101 is our virtual Florida, and Room 102 is our virtual Kansas” is not sufficient to emulate actual network-based cyberspace work and the logistical challenges and surprises that it engenders. For example, distant communication may limit bandwidth, increase latency, or simple availability of personnel because of time zone differences—factors easily overlooked when creating a synthetic environment.


The Cyber Immersion Lab is closing the gap

The USCYBERCOM J9 (Advanced Capabilities Directorate) operates the Cyber Immersion Lab (CIL), which is closing the gap between operations and research. The CIL is an adaptable, human-centered facility that develops and assesses cyberspace capabilities as well as conducts experiments to identify technology requirements for the CMF. The CIL includes five project spaces, which can accommodate between ten and twenty-five persons each, and can conduct unclassified, secret and top secret projects concurrently. The CIL’s infrastructure can support off-network projects as well as remote connections to other labs, ranges or operational networks. USCYBERCOM or adjacent research partnerships internally fund CIL projects. The core strength of the CIL is its proximity and access to the CMF in the National Capital Region.

Figure 1 – Cyber Immersion Lab

A recent illustration of the importance of experimentation in cyber operations was a project to evaluate network-mapping capabilities for operational fit in the CMF. The Network Mapping Project, which was sponsored by OUSD(I) SIGINT-Cyber, included multiple complementary investigative strategies. It began with a field study of cyber teamwork to determine evaluation conditions and criteria. It also included market research to identify candidate capabilities and culminated with an experiment that involved 20 members of 10 Cyber Protection Teams. In addition to a report that described the relative performance of the sampled capabilities, the project generated a number of other insights to inform operations as well as research communities.

For example, a key insight from the project was that network mapping, or the rendering of data about networks into a graphical format supports decision-making and information sharing.  In the past, network engineers used network maps for configuration management, property accountability, or network planning activities. In cyberspace operations, network maps must support:

  1. Shared understanding of topology, activity, and vulnerabilities,
  2. Shared information about adversary and friendly resources and actions,
  3. Military planning in cyberspace as well as physical military domains.

Because decisions in cyberspace differ across teams and echelons, so the maps to support those decisions must be commensurately varied. There is no universally reliable mapping activity, with cyberspace operations requiring federated sense-making at enterprise scale. Although the CIL network-mapping project did not identify a single existing materiel solution for the CMF, it did generate a rich description of the features that are required. More importantly, it serves as a proof of concept for experimentation to elucidate technology requirements.

Figure 2 – Cyber Team members participate in an experiment in the Cyber Immersion Lab


Campaign of experimentation

Traditional views of experiments – factorial manipulation of variables and tightly controlled circumstances – are unwieldy and under-informative in cyberspace experimentation. First, factorial designs are prohibitively resource and time intensive. For example, evaluating five tools with experienced and less experienced participants would require ten experimental conditions. Each condition would require unique, but equally representative environments to avoid the effects of learning the environment. Emergent operational conflicts challenge the sustainment of participant groups to execute such lengthy designs. Second, tight control of variables eliminates the important messiness of the real world. Isolating unpredictability and complexity of the world results in sterile, and therefore, unrealistic conditions. Cyberspace work is necessarily subject to numerous vagaries and unanticipated circumstances (e.g., the customer does not have a map of their own network; computers do not initialize efficiently, and so forth).

We recommend a ‘campaign of experimentation’ with select combinations of variables or conditions employed across a series of mini-experiments alleviating impractical factorial designs. The cyber community should collaborate in the conduct of more research to achieve the general methodological aim. Future experiments will involve multiple teams of cyber professionals located at multiple physically distributed locations, and orchestrated to conduct network analyses comparing a number of different network analysis tools, or to study different work process models. Going forward, it is crucial that USCYBERCOM and the DoD cyber community broadly, develop a robust methodology for conducting such a Campaign of Experimentation. A recent National Science Foundation report provides a similar call for empiricism to improve cybersecurity.[3]

The key to applied research and experimentation for cyberspace operations is the coalescing of researchers, developers, and the operational community in an environment that supports realistic live, virtual, and synthetic environments that are easily reconfigurable. These three groups enable the evaluation of the science, technology, human factors, engineering development and employment concepts, and the potential operational effectiveness of such research. Effective experimentation results in not only appropriate feedback for the research community but ultimately accelerates technology transition to the operational community. Without these three groups of people working together in an environment that supports loosely controlled experimentation and exploration, the ability to keep pace and advance ahead of our adversaries in the cyberspace domain will continue to be challenging as the adversary uses the Internet continuously to perform this type of research, development, experimentation, and technology transition.

Disclaimer:  This paper reflects the views of the authors.  It does not represent the position of the Department of Defense or US Cyber Command. 


[1] R. Hoffman, and K. Deffenbacher, “A multidimensional analysis of the relations of basic and applied psychology”. Theoretical Issues in Ergonomic Science. (2011) DOI: 10.1080/1464536X.2011.573013

[2] R. Hoffman, G. Lintern, and S. Eitelman, (March/April 2004). “The Janus Principle. IEEE: Intelligent Systems, (March/April 2004), 78-80.

[3] D. Balenson, L. Tinnel, L., T. Benzel, “Cybersecurity Experimentation of the Future (CEF): Catalyzing a New Generation of Experimental Cybersecurity Research.  National Science Foundation. (2015) at: (accessed on 1 March 2016).

US Army Comments Policy
If you wish to comment, use the text box below. Army reserves the right to modify this policy at any time.

This is a moderated forum. That means all comments will be reviewed before posting. In addition, we expect that participants will treat each other, as well as our agency and our employees, with respect. We will not post comments that contain abusive or vulgar language, spam, hate speech, personal attacks, violate EEO policy, are offensive to other or similar content. We will not post comments that are spam, are clearly "off topic", promote services or products, infringe copyright protected material, or contain any links that don't contribute to the discussion. Comments that make unsupported accusations will also not be posted. The Army and the Army alone will make a determination as to which comments will be posted. Any references to commercial entities, products, services, or other non-governmental organizations or individuals that remain on the site are provided solely for the information of individuals using this page. These references are not intended to reflect the opinion of the Army, DoD, the United States, or its officers or employees concerning the significance, priority, or importance to be given the referenced entity, product, service, or organization. Such references are not an official or personal endorsement of any product, person, or service, and may not be quoted or reproduced for the purpose of stating or implying Army endorsement or approval of any product, person, or service.

Any comments that report criminal activity including: suicidal behaviour or sexual assault will be reported to appropriate authorities including OSI. This forum is not:

  • This forum is not to be used to report criminal activity. If you have information for law enforcement, please contact OSI or your local police agency.
  • Do not submit unsolicited proposals, or other business ideas or inquiries to this forum. This site is not to be used for contracting or commercial business.
  • This forum may not be used for the submission of any claim, demand, informal or formal complaint, or any other form of legal and/or administrative notice or process, or for the exhaustion of any legal and/or administrative remedy.

Army does not guarantee or warrant that any information posted by individuals on this forum is correct, and disclaims any liability for any loss or damage resulting from reliance on any such information. Army may not be able to verify, does not warrant or guarantee, and assumes no liability for anything posted on this website by any other person. Army does not endorse, support or otherwise promote any private or commercial entity or the information, products or services contained on those websites that may be reached through links on our website.

Members of the media are asked to send questions to the public affairs through their normal channels and to refrain from submitting questions here as comments. Reporter questions will not be posted. We recognize that the Web is a 24/7 medium, and your comments are welcome at any time. However, given the need to manage federal resources, moderating and posting of comments will occur during regular business hours Monday through Friday. Comments submitted after hours or on weekends will be read and posted as early as possible; in most cases, this means the next business day.

For the benefit of robust discussion, we ask that comments remain "on-topic." This means that comments will be posted only as it relates to the topic that is being discussed within the blog post. The views expressed on the site by non-federal commentators do not necessarily reflect the official views of the Army or the Federal Government.

To protect your own privacy and the privacy of others, please do not include personally identifiable information, such as name, Social Security number, DoD ID number, OSI Case number, phone numbers or email addresses in the body of your comment. If you do voluntarily include personally identifiable information in your comment, such as your name, that comment may or may not be posted on the page. If your comment is posted, your name will not be redacted or removed. In no circumstances will comments be posted that contain Social Security numbers, DoD ID numbers, OSI case numbers, addresses, email address or phone numbers. The default for the posting of comments is "anonymous", but if you opt not to, any information, including your login name, may be displayed on our site.

Thank you for taking the time to read this comment policy. We encourage your participation in our discussion and look forward to an active exchange of ideas.