Farmington, Connecticut–based United Technologies Corporation (UTC) provides high-technology systems and services to the building and aerospace industries worldwide. The corporation’s research and innovation arm, United Technologies Research Center (UTRC), supports the development of new technologies and capabilities across UTC with the goal of solving pressing needs in fields represented by the company’s holdings.
A long-established focus for UTRC lies in combustion research for jet engines manufactured by UTC business unit Pratt & Whitney, one of the world’s largest aircraft engine manufacturers. UTRC uses a combination of experimentation and computational modeling in its research. However, combustion processes are very difficult to interrogate experimentally, requiring researchers to increasingly rely on computational modeling.
“UTRC has a long-standing commitment to investigating what goes on inside the combustor,” said Vaidya Sankaran, principal engineer in combustion science at UTRC, referencing the part of a jet engine that houses the combustion chamber and fuel injector.
“The problem is, it is very hard to ‘see’ inside due to limited access and the adverse temperature and pressure conditions that exist there. But you can visualize what is happening with a computer simulation.”
In an effort to study the complex fluid dynamics and chemical reactions occurring inside a jet engine combustor, UTRC researchers teamed up with experts at the Oak Ridge Leadership Computing Facility (OLCF), a US Department of Energy (DOE) Office of Science User Facility located at DOE’s Oak Ridge National Laboratory.
By identifying more comprehensive methods of modeling the chemical processes taking place inside a combustor, UTRC researchers will be able to develop cleaner, safer, and more efficient aircraft engines and thereby increase their competitiveness in the industry. In addition, as new jet fuels are developed, UTRC will be able to simulate these fuels’ differences across multiple engine models and operating conditions, thus reducing the number of expensive tests required to validate their research results.
Combustion’s complexity
The design of next-generation jet engines requires evaluation of a multitude of processes at the most fundamental level.
“If you simulate these processes reliably, you can actually visualize them to derive insights,” Sankaran said. “But the issues related to fluid dynamics, thermodynamics, chemical kinetics, and heat transfer, among other factors, are varied and extremely complex.”
Indeed, these simulations are so complex that even state-of-the-art computers are unable to simulate all the processes taking place simultaneously throughout an entire combustor. So instead of retaining all the complexities of the combustor, only those features that are needed to study mixing and combustion are retained and simulated.
Jerry Lee, UTRC staff engineer in propulsion technology, noted that modeling a smaller cross section of the combustor enables researchers to run more meaningful simulations. “In terms of fluid mechanics, what’s happening in gas-turbine combustors is turbulent combustion,” he said. “Modeling this is an intractable problem because the flow has multiple scales in time and space, and most can’t be resolved with computers for the foreseeable future. Yet, if we model a limited number of relevant scales, a previously intractable problem becomes much more tractable.”
To model combustion, researchers rely on two major methods: direct numerical simulation (DNS) and large eddy simulation (LES). Each method has its points of distinction, Sankaran explained, likening the two approaches to the painting styles of the world’s great masters. “When you think of Rembrandt’s paintings, they are highly detailed, very realistic, almost photographic in nature.
“However, if you look at Van Gogh’s scenes, the style is much coarser. You still know exactly what you’re looking at, but it may not be fine-grained like the Rembrandt. In our work the DNS simulations are the Rembrandts, and the LES simulations are the Van Goghs.”
Although UTRC researchers needed only Van Gogh-level detail in their production simulations, they also needed to run Rembrandt-like DNS calculations to improve the models being used for the LES runs.
DNS calculations are very computationally intensive and typically require high-performance computing (HPC) facilities like those available in national laboratories to complete the complex calculations within a reasonable timeframe. Realizing that a large-scale supercomputer with accelerators (also called GPUs) running in concert with traditional CPUs would enable the team to clear some of its computational hurdles, the UTRC team turned to the OLCF and its Cray XK7 Titan supercomputer, the most powerful in the nation for open science, with its heterogeneous architecture that combines the capabilities of CPUs and GPUs.
LES is more
Sankaran’s team received a Director’s Discretionary allocation of time on Titan through the OLCF’s industrial partnership Accelerating Competitiveness through Computational ExceLlence (ACCEL) program. ACCEL helps companies gain access to OLCF resources for high-impact, complex problems that exceed a firm’s internal computing capabilities.
At the OLCF the team pursued two critical goals. The first was to accelerate, via GPUs, the team’s proprietary, modified version of the Georgia Institute of Technology’s Large Eddy Simulation with Linear Eddy (LESLIE) code. UTRC researchers sought to accelerate LESLIE’s performance by running the chemical kinetics calculations in the code on GPUs. Using the DNS approach, the team performed development, implementation, and testing of GPU acceleration algorithms by simulating a simpler problem with complex chemical reactions.
UTRC’s second goal was to perform a cost-benefit analysis of a well-understood, but rarely used, method of LES calculation called explicitly filtered LES.
When researchers perform LES, they typically rely on implicitly filtered LES methodology. Because of the inherent assumptions in this approach, the errors stemming from numerical models and physics models interact in an unpredictable manner that can, in many cases, lead to unreliable results. The simulation results reveal features and detail that researchers like, but they also include a higher degree of error than those generated by explicitly filtered LES runs. However, few researchers attempt the more accurate explicitly filtered LES calculations because of the enormous computing resources required to perform them.
UTRC was interested in answering the following questions: Would the more mathematically rigorous explicitly filtered LES calculations produce substantially more accurate results to justify the dramatic increase in computing needed to achieve them? Would it be worth the investment for the new insights and competitive edge UTRC might achieve?
Predictive simulation capabilities, coupled with smart testing and experimentation, are key to improving current designs and developing next-generation combustors with increased efficiencies and reduced pollutants. This exercise allowed UTRC to explore and evaluate a potentially accurate modeling methodology and conduct a cost-benefit analysis. To the best of the researchers’ knowledge, this is the first time such a cost-benefit analysis has been systematically performed for reacting flows. “Without OLCF’s computing resources, such intensive computations are not possible using UTRC’s internal computing resources only,” said Sankaran.
New insights, new decisions
By gaining access to Titan, the UTRC team was able to successfully port its modified LESLIE code to run on GPUs as well as CPUs. This not only enabled the researchers to run test problem simulations with improved chemical kinetics; it also gave them a fivefold increase in computational speed compared with CPU-only methods. “We were able to simulate the problem, with detailed chemical kinetics, in a reasonable amount of time,” Sankaran said. “That allowed us to explore what happens at the extremely fine scales you can’t easily see when conducting experiments.”
Lee added, “Without the GPUs on Titan, we would not have been able to complete the DNS calculations for our research.” OLCF scientific liaison Ramanan Sankaran played an important role in helping the team run the code efficiently on Titan’s GPUs.
In addition, the team successfully completed its comparison of the implicitly filtered LES calculations with the more accurate explicitly filtered simulations. “For UTRC the gain in accuracy was not worth the cost of the additional computing resources needed to achieve it,” said Sankaran. “It may have looked obvious in hindsight, but so few people use explicitly filtered LES, and just as rarely for the combustion simulations that are critical to our research, so we had to test this for ourselves. Titan enabled us to do this important cost-benefit analysis.”
Sankaran also explained that “until a better approach can be identified, UTRC and the larger LES community have to continue using implicitly filtered LES, relying on our experience to smartly employ the approach to avoid any inherent pitfalls.” And while explicitly filtered simulations are too computationally expensive to perform regularly, the ability to conduct them could help propel UTRC to the vanguard of combustion modeling and lay the groundwork for improving its simulations as computational power continues to increase.
Both Sankaran and Lee noted UTRC achieved additional benefits by working on Titan. “Access to Titan gives us a chance to evaluate large-scale, heterogeneous architectures and reduces our risk when making future HPC investment decisions,” said Sankaran. “This experience will help guide us as we develop new codes. It also made us aware of the different expertise we need internally to use these systems effectively.”
For now UTRC is closely monitoring the upward arc of GPUs in the HPC industry and projecting future competitive needs. UTRC’s experience with Titan positions the company to act—rather than react—in light of industry trends. “Now with any new development work we do, we keep heterogeneous architectures in mind,” Sankaran said.
Related Publication: Peter A. Cocks, Vaidyanathan Sankaran, and Marios Soteriou, “Towards Predictive Reacting Flow LES.” 52nd Aerospace Sciences Meeting, AIAA SciTech Forum, (AIAA 2014-0826), https://dx.doi.org/10.2514/6.2014-0826.
Oak Ridge National Laboratory is supported by the US Department of Energy’s Office of Science. The single largest supporter of basic research in the physical sciences in the United States, the Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.
