Mar 9 2016
A few years ago, a team of international scientists parlayed many years of research into the discovery of the Higgs boson, a subatomic particle, thought to be a building block of our universe. The analysis of data collected from several particle collisions was aided by HTCondor, a software program.
Miron Livny
Now HTCondor is helping researchers to identify gravitational waves that were caused 1.3 billion years ago due to the collision of two black holes, each 30 times bigger than the Sun.
From revealing the Higgs boson, one of the smallest particles in science, to identifying the huge astrophysics of black holes, the HTCondor High Throughput Computing (HTC) has become indispensable for processing complex and large data generated by international science. Computer scientists from the University of Wisconsin—Madison have pioneered the distributed high throughput technologies for the past 30 years.
In February an announcement was made that scientists at the Laser Interferometer Gravitational-Wave Observatory (LIGO) have unraveled Albert Einstein’s Theory of Relativity, and proven that gravitational waves generate ripples through time and space. This discovery too involves the HTCondor. Since 2004 the program has been integral to data analysis portion of the project, which involves over 1,000 scientists from 80 organizations from 15 countries.
For the past 12 years, over 700 LIGO scientists have used the HTCondor software for complex data analysis on computers in Europe and the U.S. Approximately 50 million core-hours were managed by HTCondor during the last six months for the data analysis efforts.
Two Black Holes Merge Into One
HTCondor’s road to LIGO was the result of a collaboration that began about 10 years ago between two University of Wisconsin (UW) teams: a 29-member LIGO group at UW, Milwaukee, and the HTCondor team at UW, Madison. This collaboration helped to change to LIGO computational model to the advanced version of the HTC.
Miron Livny, a UW–Madison professor of computer science and the chief technology officer for the Morgridge Institute for Research and the Wisconsin Institute for Discovery, heads the HTCondor team. The HTCondor program uses novel high-throughput computing technologies to exploit the capabilities of thousands of computer networks to run large ensembles of computational tasks.
What we have is the expertise of two UW System schools coming together to tackle a complex data analysis problem. The problem was, how do you manage thousands upon thousands of interrelated data analysis jobs in a way that scientists can effectively use? And it was much more easily solved because Milwaukee and Madison are right down the street from each other.
Thomas Downes, Senior Scientist of Physics, UW-Milwaukee
The HTCondor was being used by the UW–Milwaukee group since the early 2000s for the NSF Information Technology Research (ITR) project. Its then-lead scientist, Bruce Allen, moved to take over as the director of the Albert Einstein Institute for Gravitational Physics in Hannover, Germany, a leading center for the LIGO project. Duncan Brown, then a UW–Milwaukee physics Ph.D. candidate, became a professor of physics at Syracuse University, and led its LIGO efforts.
Brown, Allen and others worked at the Milwaukee for demonstrating the ability of the HTCondor technique to the LIGO mission, subsequently resulting in its adoption at other LIGO sites. HTCondor became the sought-after technology for the main LIGO data teams at the UW–Milwaukee, the Albert Einstein Institute, Syracuse University, the Cardiff University in the UK, and the California Institute of Technology (Caltech).
Peter Couvares has an all-round view of HTCondor’s links with LIGO. He has been working with the HTCondor team for a decade at the UW–Madison, and managed the links between HTCondor and LIGO for five years after moving to Syracuse to join the LIGO team led by Brown. Now, Couvares manages the LIGO data analysis computing group and is a senior scientist at Caltech.
Why the HTCondor program is considered as a boon to big science efforts such as the LIGO?
“We know it will work — that’s the killer feature of HTCondor,” says Couvares. It works, he says, since HTCondor is geared up for the key challenge of distributed computing. It is not possible to visualize a network comprising thousands of individual computers will not face local faults. This possibility is built into the core of the HTCondor.
The HTCondor team always asks people to think ahead to the issues that are going to come up in real production environments, and they’re good about not letting HTCondor users take shortcuts or make bad assumptions.
Peter Couvares, Senior Scientist, Caltech
This type of approach is crucial for a project like LIGO. The constant data stream from LIGO detectors comprise gravitational information, seismic activity, light, temperature and wind, which help to differentiate between good data and bad data.
In the absence of noise, this would have been a very easy search, but the trick is in picking a needle out of a haystack of noise. The biggest trick of all the data analysis in LIGO is to come up with a better signal-to-noise ratio.
Peter Couvares, Senior Scientist, Caltech
Stuart Anderson, LIGO senior staff scientist at Caltech, has been promoting the application of the HTCondor in LIGO for more than 10 years. According to him, it is more of human element than so much about technology that makes HTCondor to succeed.
The HTCondor team provides a level of long-term collaboration and support in cyber-infrastructure that I have not seen anywhere else. The team has provided the highest quality of technical expertise, communication skills and collaborative problem-solving that I have had the privilege or working with.
Stuart Anderson, LIGO Senior Staff Scientist, Caltech
Adds Todd Tannenbaum, the present HTCondor technical lead who works in close association with Couvares and Anderson: “Our relationship with LIGO is mutually profitable. The improvements made on behalf of our relationship with LIGO have greatly benefited HTCondor and the wider high throughput computing community.”
In 2001, Ewa Deelman, a research associate professor and research director at the University of Southern California Information Sciences Institute (ISI), became associated with HTCondor, when she introduced Pegasus, which is a system that carries out workflow automation for the benefit of scientists employing systems such as HTCondor. Jointly, HTCondor and Pegasus help the scientists to cross the technological barrier.
I think the automation and the reliability provided by Pegasus and HTCondor are key to enabling scientists to focus on their science, rather than the details of the underlying cyber-infrastructure and its inevitable failures.
Ewa Deelman, Research Associate Professor, University of Southern California Information Sciences Institute
The future of LIGO looks exciting, and the basic high-throughput computing technologies of the HTCondor will undergo changes along with the changes in computing technologies and science. Many scientists concur that the initial observation of the gravitational waves will usher in huge amount of data about the universe in the future.
“The field of gravitational wave astronomy has just begun,” says Couvares. “This was a physics and engineering experiment. Now it’s astronomy, where we’re seeing things. For 20 years, LIGO was trying to find a needle in a haystack. Now we’re going to build a needle detection factory.”
What started 15 years ago as a local Madison-Milwaukee collaboration turned into a computational framework for a new field of astronomy. We are ready and eager to address the evolving HTC needs of this new field. By collaborating with scientists from other international efforts that use technologies ranging from a neutrino detector in the South Pole (IceCube) to a telescope floating in space (Hubble) to collect data about our universe, HTC will continue to support scientific discovery.
Miron Livny, Professor of Computer Sciences, UW-Madison