Modern software development for aerospace

Credit: scyther5 | Thinkstock.com

The role of traditional information technology (IT) within organizations is being challenged by consumerization. Today’s enterprises are being pushed to provide experiences that rival those provided by consumer companies – users’ expectations are driven by the devices and technologies they use in their personal lives. Those expectations are carrying into the design, speed, and functionality of enterprise products that hardly ever considered end-user needs.

Aerospace is an industry feeling the effect of this shift, and it needs to change. But how?

The aerospace industry can learn from tried and true software development methodologies to adapt and modernize aerospace manufacturing environments to increase operational efficiency and lower production costs while providing customers with a superior product to drive consumer satisfaction.

Continuous delivery

Continuous delivery is an approach to software engineering, designed to build, test, and deploy software more frequently for improved user experience. The process attempts an optimal blend of the human creative element of software development with the efficiency of manufacturing processes. Continuous delivery starts when a human completes creative work. This hand-off from human to computer is achieved when a programmer commits an incremental software change to a version management system. The version management system tracks the author and contents of each change, making it easy to see all changes.

From that point, all remaining tasks necessary to deliver a software product get automated. Software product builds, deployments to test environments, and execution of various simulations are executed frequently, potentially many times a day. As far as the release process and infrastructure are concerned, any given software change could be the one that results in an official release to customers.

In typical aerospace and manufacturing processes, the majority of deliveries are strictly to internal consumers, such as quality assurance. External releases tend to occur rather infrequently, perhaps with milestones taking 18 months. Continuous delivery may be met with a “not for us” reaction as the logical extreme best suits environments where the need for making changes fast is great and the cost of failure is low. When continuous delivery is applied in processes with a premium on initial quality, the benefit comes from lowering the cost of getting software changes into an environment where they can be tested quickly.

With the methodology, project status dashboards show each software component or module as a happy or angry icon (e.g. green/red dots). Staff are trained to treat any unhappy icons on the dashboard with a sense of urgency. A full green dashboard for all components is required to achieve delivery.

When a human selects a change to be the last one prior to a release, the process for delivering that change will have been executed for the 10,000th time, not the first. Continuous delivery processes provide more than efficiency; they build confidence. That’s especially critical in processes when there are few, or perhaps even a single opportunity for an actual release.

Continuous delivery should be applied pragmatically, automating everything within reason, while a few manual processes serve to elevate particular deliveries above the steady hum of continuous deliveries. This includes things that defy easy automation, or that take too long to be integrated into a continuous daily cycle. Some examples are:

• Creative manual and exploratory testing – What happens if I try this?
• Regression test suites to confirm that, while implementing new functionality, you didn’t break anything that used to work. Large numbers of regression tests accumulate throughout time, so an entire regression test suite for a mature product tends to be long-running
• Load and performance tests determine how software performs under extreme conditions. By the very nature of what they test, these tend to be resource- intensive and long-running
• Generation of formal release notes suitable for external consumption
• Simulators, which tend to be resource-intensive and involve manual setup

Achieving continuous delivery requires an investment in release engineering processes and tooling. It also promotes communication and coordination among various organizations involved in the release process, such as development, quality assurance, release engineering, and operations.

Many studies confirm the dramatic cost savings and return on investment (ROI) in release engineering and continuous delivery processes. It is well known in the software development world that, just as with manufacturing, the cost of finding and fixing problems differs by orders of magnitude depending on which phase in the process the fix occurs.

The amount of rework and associated costs required goes up the longer it takes to find and fix problems. Continuous delivery processes tighten cycle time and help drive fast feedback, reducing costs, and eliminating human error in processes that are best done by machines.

DevOps

Where continuous delivery emphasizes automation of mechanical processes and tools, the collection of practices and trends known as DevOps goes further, emphasizing a broader scope and tighter collaboration among people in certain development and operational roles. Continuous delivery is a step along the way to a complete DevOps pipeline and cultural transformation.

Component-based development (CBD)

Credit: maciek905 | Thinkstock.com

Large scale, pure software development projects (those not related to hardware components) and embedded systems projects at any scale often practice some form of component-based or modular development. With embedded systems, the very nature of working with hardware tends to drive development in clearly defined components.

Pure software systems often grow organically, starting small and eventually becoming large enough to be referred to as monolithic systems. Monolithic systems tend to defy the fast build, test, and deployment cycles demanded by continuous delivery. At some point of scale, the benefits of a modular architecture (such as being able to replace or add any component without affecting the rest of the system) tend to outweigh the initial simplicity of systems that are completely interwoven, monolithic systems. Conversion from monolithic to modular architectures is a common and recurring theme in the software industry.

Seasoned developers and development organizations prefer to avoid building monoliths to start. As with other industries, the aerospace industry has matured in application of CBD principles, investing in the planning and discipline to encourage development of modular systems by initial design rather than expensive conversion from a grew-too-big monolithic system.

Assembling software products from many components requires investment in locally appropriate development standards, build, and possibly dependency management systems, and in some cases code frameworks. It also entails a bit of a mindset shift, as developing components promotes better re-use from the start. The difference is subtle: when developing a module that is part of a large monolithic system, less attention is paid to future re-use of that component than when it is developed initially.

The benefit of building for re-use is limited if a single airplane is built but becomes dramatic when multiple aircraft benefit from the same technology. This engineering for reusability matters even in those aerospace programs where only a relatively small number of planes are built. In such cases, each one tends to be a high value item, and the component is extensively tested. Re-using a well-tested, well-proven module helps lower costs.

Advances in CBD

CBD concepts have been around for a long time, but modern advances are improving several aspects of CBD, such as version management practices and life cycle management.

Version management is a component of configuration management that tracks revision history for documents, programs, code, or files of any kind. The most basic version management challenge of CBD is complex dependency management. Typically, a set of software products are built from a different set of components, with each product using different versions of the same pool of components. Each component may, in turn, depend on other components, with several layers of software between the foundation components and top-level products. Better version management solutions help identify situations where multiple versions of a component are inadvertently used in the same product, known as diamond dependency.

Perhaps the biggest challenge of CBD, and the one most matured, is the management of atomic changes in such large-scale modular architectures. Organizations managing large-scale modular systems have proven methods for safely making coordinated, broad-impact changes. The approach relies on having a single source of truth, a monorepo – a single repository capable of tracking changes to a set of products and all components used to build those products. A monorepo simplifies the search for affected components and allows change to all layers to be made as an atomic transaction, making it easier to evaluate the impact and cost of a single software change (and making it easier to revert the change should that become necessary). Upon the commit, continuous delivery processes quickly test the impact of the change at all levels, including low-level components and all impacted products.

Software life cycle management has also improved in recent years. In addition to tracking changes to software code, managing a product’s configuration is also necessary. At its simplest, a configuration entails the list of versions of each component used. Tracking changes to both code and configuration is necessary with CBD. Configuration changes sometimes drive code changes, and vice-versa.

Mature solutions provide environment management systems, applying configuration management principles to manage and track which components are deployed and what resources (hardware, storage systems, databases, etc.) are available.

Code reviews

Those valuable conversations have relevant context, and can easily be discovered years later. The review process makes mistakes less likely to happen, and can provide valuable information when trying to find out why a software change was made that led to a system failure or product recall. Code review processes have collateral benefits as well, such as cross-training.

© Simfan | Dreamstime.com

Pre-commit code review

Basic code review starts with post-commit review processes, which start after code is committed to a version control repository. Some organizations use advanced, pre-commit code review processes, intended (for example) to ensure that nothing ever gets into the version control system that hasn’t at least had a second set of eyes on it. Typically, pre-commit code review processes are required only for the most highly critical modules. In less-critical modules, pre-commit reviews are sometimes used as a training tool, where a junior developer might make a proposed change and request a review by peers or a tech lead prior to committing to the repository.

Test-driven development

Test-driven development (TDD) is complementary with continuous delivery and CBD. Essentially, TDD reverses the traditional approach. Traditionally, an engineer would develop a module, and then wait for someone else maybe later to write a test for it. TDD is a more disciplined approach that clarifies the developer is also responsible for writing unit tests. It promotes that tests be written before functional software. The first of the continuous builds of a module executes a test with only a stub for the new module, and therefore always fails.

As an extension, the first change to the functional software is to add documentation, creating a doc-only version that says what it will do before it does anything.

In aerospace, TDD can be expanded beyond unit tests. Complex modules that interact with hardware systems may be prohibitively expensive for actual testing. Such modules are tested with simulators that inject environmental stimuli to the software component, and then capture and evaluate outputs. Simulators tend to be sophisticated software products, and are often developed in parallel with the systems they are intended to test.

Version everything

Aerospace software projects produce more artifacts than software source code. Versioning is best applied throughout the software life cycle to track the many items that evolve over time. Versioned items typically include requirements documents, marketing documents, source code, test suites and simulators, fully built software, and more. If it evolves, version it.

With this breadth of modern software development practices, aerospace and manufacturing environments can greatly improve their speed, reduce costs, and provide a product which will beat any modern expectations.

Perforce Software

www.perforce.com

About the author: C. Thomas Tyler is a principal solutions consultant and developer evangelist at Perforce Software. He can be reached at ttyler@perforce.com.