These days, no computer is an island. High-performance networks enable computer centers to share resources and communicate with users. Grid computing would be impossible without them. Even PC users feel lost without an Internet connection. But high-performance networks can be temperamental creatures, prone to such maladies as traffic jams, routing inefficiencies, and outright link failures. They need constant attention from network system engineers to run at peak performance. But how can you find the source of a problem in a networked system with dozens of sites when all of their administrators are convinced that the problem must be someone else's fault?

The Measurement and Network Analysis group of the National Laboratory for Applied Network Research (NLANR/MNA) assesses the performance of next-generation computer networks -- measuring the flow of message traffic, analyzing performance issues, and making all of the data, analyses and tools available to the community -- so they can be tuned for maximum end-to-end performance. In particular, network measurements are essential for identifying and locating problems (malfunctions, bottlenecks, inefficiencies, incompatibilities, etc.) in ultrafast research networks and in high-speed international links.

"Although NLANR's measurement and analysis efforts may not be visible to casual users of the Internet, all users ultimately benefit," said Tom Greene, senior program director for the National Science Foundation's Division of Advanced Networking Infrastructure and Research. "Measurement and analysis tools are essential for solving performance issues, and we all need solutions there."

Based at the San Diego Supercomputer Center (SDSC), the NSF-funded NLANR/MNA group has created a network analysis infrastructure to conduct their own research and to support the efforts of outside researchers, systems administrators, and students. With few exceptions, the monitoring devices have been located on high-performance networks operating in the United States. International expansion began cautiously three years ago; now the research infrastructure extends to other continents, and network researchers and system administrators around the world are enthusiastically collaborating with NLANR/MNA researchers, setting up measurement and analysis research projects of their own.

Gathering Data

NLANR/MNA research has two main components. The Passive Measurement and Analysis (PMA) project uses information gleaned from observing network traffic, without interacting with the networks themselves. A low-technology analogy for this project would count envelopes, weigh them, and sort them by postmark to determine how efficiently the U.S Postal Service is handling the mail. The PMA project uses network message packet header traces instead of envelopes, but the principle is the same.

The Active Measurement Project (AMP) performs site-to-site active measurements and analyses, which enable network researchers and engineers to track problems and changes in network performance, by inserting test messages into the networks it studies and observing their progress through the systems. A Postal Service analog of the AMP project would send test letters between many locations, noting their travel times and postmarks. More than 130 AMP monitors are deployed on high-speed research networks in the United States and in other countries, most of them in a "full mesh" configuration -- each monitor exchanging messages with every other monitor -- to measure message path, round-trip time, and packet loss. In addition to gathering data, the AMP project also analyzes long-term, large-scale trends in message flow patterns.

The monitoring devices for both projects are rack-mounted PCs with high-speed network cards, installed in the network equipment racks of universities and research institutions that participate in the two projects. All participation is voluntary, and both NLANR/MNA and participating sites are meticulous about maintaining the privacy of network users. To use another Postal Service analogy, the monitors count envelopes and examine postmarks and Zip codes, but they do not look at the full addresses or open any of the envelopes or read the letters.

"We go to a lot of trouble to anonymize any address information before we ever see it," said Hans-Werner Braun, founder of NLANR and principal investigator of the Measurement and Network Analysis group. "It's like encrypting the Zip codes with a key that we don't have -- it's non-reversible, so we can't tell what the addresses really were, but we use the anonymized versions of the addresses for statistical purposes."

International Networks

Four years ago, researchers in Norway, New Zealand, and Israel borrowed single network monitor units from NLANR/MNA on an experimental basis. International interest in cooperative network research was low-key, until one small event started a landslide.

The NSF created the High Performance International Internet Services (HPIIS) Program in 1997 to provide international researchers access to high-performance networks in the United States and to give U.S. educators access to researchers and international high-performance networks around the globe. HPIIS project participants include AMPATH (which links most of the countries in Central and South America to the U.S.), Euro-Link (between the U.S. and Europe), the TransPAC link (between the U.S. and Japan), NaukaNet (formerly called MIRnet, between the U.S. and Russia), and NSF's STAR TAP and StarLight international access points.

As a result of hosting an NSF-funded HPIIS workshop at SDSC in 2001, NLANR/MNA is now conducting joint research with all four HPIIS grant projects. The SDSC researchers also are collaborating with three additional countries not in the HPIIS program -- Australia, Korea, and Canada -- that sent representatives to the workshop.

"The entire conference cost less than $6,000," said Ronn Ritke of SDSC, the co-principal investigator and manager of the NLANR/MNA program. "But the return on investment was enormous. Thirty researchers, representing organizations that included about two dozen different countries, met for face-to-face discussions of issues. Suddenly everyone saw the advantages of international cooperation and coordinated research. The enthusiasm was contagious."

New AMP monitors were placed in Australia, Canada, and Korea only a few weeks after the conference. The NLANR/MNA group now has AMP machines in Australia, Brazil, Canada, China, Finland, Germany, Holland, Hungary, Israel, Japan, Korea, Mexico, New Zealand, Norway, and Thailand; Russia, Taiwan, and China soon will host monitors as well. They already had placed a PMA monitor in Israel, and soon there will be PMA monitors on networks in Thailand, Germany, and Korea. The STAR TAP and StarLight international links include both AMP and PMA monitors, and many measurement machines are located at individual U.S. sites that serve as international entry/exit points on high-speed networks. And as part of the cooperative effort, all of these projects are sharing their data with other researchers around the world.

The result is that NLANR/MNA is well on the way to establishing a world-wide infrastructure for gathering performance data from high-speed next-generation research networks, vitally important in an age of global communications.

But of equal importance is the intangible network of person-to-person contacts among scientists and engineers in dozens of countries that have been engendered by the program. Researchers and administrators are engaging in joint efforts, exchanging information, and helping to solve one another's problems using a range of different collaboration models, all started as experimental efforts facilitated by NLANR/MNA's activities.

The NLANR/MNA participates in cooperative network measurement research efforts with Australia, Brazil, Canada, China, Finland, Germany, Holland, Hungary, Israel, Japan, Korea, Mexico, New Zealand, Norway, Russia, Taiwan, and Thailand. Click here for a full-size map.

The success of the effort can be judged by another workshop that NLANR/MNA hosted at SDSC less than two years after the HPIIS meeting. PAM2003, the international passive and active network measurement conference, was a three-day event attended by more than 80 researchers and students from Australia, Belgium, Canada, Croatia, France, Germany, Italy, Japan, the Netherlands, Norway, the United Kingdom, and Sweden as well as from the United States. A record number of papers was submitted to the workshop -- of 92 submissions, 20 were accepted as full papers and five were accepted as posters.

Exactly 40 of the attendees were graduate students. "One of our major goals was to promote student participation," Ritke said. "It's important to cultivate the next generation of network researchers. A mixed workshop like this one enables us to encourage the best and the brightest students to go into the field and gives them a head start on working with well-established researchers. The students not only attended, they were full participants -- 12 of the 20 accepted papers listed students as first authors."

Technology transfer is an important aspect of the collaborations. AMP technology is now being used in parallel national programs to instrument and characterize the performance of the domestic research networks of Australia, Korea, and Taiwan; interest in building similar national infrastructures has been expressed by research groups in Thailand, Brazil, Russia, and China.

And the transfer flows in more than one direction -- these cooperating national efforts are being integrated into the infrastructure established and run by NLANR/MNA, with new ideas shared among all participants. For example, presentation topics by international participants at PAM2003 included leading-edge research in active measurement (sending test packets and observing their progress through the network), passive measurement (observing actual traffic on "live" networks), performance metrics (developing indicators to characterize traffic behavior), traffic statistics (to develop models of "real" network traffic), and visualization (finding effective ways to display what is happening in a network).

"The continuous growth of the NLANR Measurement and Network Analysis effort has fostered the growth of wider understanding of the importance of network measurement and the international demand for collaborations with the NLANR team," said Tom Greene of NSF.

The scope of the NLANR/MNA group's international activities is illustrated by the following examples.

Europe

Euro-Link is an NSF-funded initiative to connect European and Israeli national research networks to American networks, supported through the HPIIS program as a next-generation Internet initiative for international research collaboration. Euro-Link consortium members NORDUnet, SURFnet, RENATER2 and CERN, along with partner HPIIS network consortia TransPAC and MIRnet, connect to the U.S. through the Science, Technology and Research Transit Access Point (STAR TAP). StarLight, the optical STAR TAP initiative, is an advanced infrastructure and proving ground for services on high-speed (gigabit and 10-gigabit) optical networks. StarLight is run by the Electronic Visualization Laboratory at the University of Illinois at Chicago, the International Center for Advanced Internet Research (iCAIR) at Northwestern University, and the Mathematics and Computer Science Division at Argonne National Laboratory, in partnership with Canada's CANARIE and Holland's SURFnet programs.

StarLight and STAR TAP are the world's most strategic international access points, with an enormous volume of international traffic. An NLANR/MNA AMP monitor is located at the SBC GigaPop in Chicago on the StarLight HPC connection. A PMA device is located at the STAR TAP GigaPop and takes passive measurements on the HPC link; passive optical taps already have been pre-installed at StarLight to avoid having to take down the operational link when NLANR/MNA's PMA monitors are installed and deployed in the near future.

Latin America

The AmericasPath project, AMPATH, began as a collaboration between Florida International University (FIU) and Global Crossing, and is now supported in part by grants from the NSF. Using Global Crossing's land-based and undersea optical fiber network, AMPATH interconnects the research and education networks in South and Central America, the Caribbean, and Mexico to U.S. and foreign networks via the high-speed Abilene network backbone of the Internet2 consortium. Participants also utilize the StarLight and STAR TAP connection points.

NLANR engineers worked with Eric Johnson, the network engineer for AMPATH, and with Julio Ibarra and Heidi Alvarez of FIU to install AMP and PMA monitors in the network access point in Florida. The active monitor is providing data, and the passive monitor is sending traffic traces that will be made publicly available.

An AMP monitor is in place at the Network Operations Center of Corporacion Universitaria para el Desarollo del Internet (CUDI) in Mexico City.

AMPATH has implemented a connection to the Academic Network at São Paulo (ANSP), the network of the State of São Paulo in Brazil. A research group in Brazil has recently installed an AMP monitor on the link from the AMPATH connection in Florida to Brazil.

Russia

RELARN is a Russian voluntary association of scientific and educational organizations, initiated in 1992 by the Ministry of Science, the Russian Academy of Sciences, and the Kurchatov Institute research center. RELARN supports non-commercial information exchange using computer networks technologies, coordinates projects on computer networks for science and education development, promotes interaction with research computer networks in Russia and abroad, and facilitates scientific contacts by organizing conferences, competitions, and exhibitions.

NaukaNet delivers enhanced international network services to Russia's emerging high-performance networks and gives them access to American researchers by connecting optical networks in the Moscow and St. Petersburg areas to the U.S. network infrastructure via STAR TAP. It links numerous collaborative projects in fields ranging from high-energy physics and the control of nuclear materials to geological studies of the earth's crust and environmental engineering. The NaukaNet initiative began with HPIIS support to the University of Tennessee, Knoxville, with matching support from the Ministry for Science and Technology of the Russian Federation.

NLANR/MNA co-principal investigator Ritke attended and gave presentations on measurement activities to RELARN meetings held in Russia in early June 2002 and June 2003. Tony McGregor, manager of the AMP project, also attended and presented at the June 2003 RELARN meeting.

"I don't think we could have brought a more representative delegation [of] the U.S. communities in science, education, grid development and networking," said fellow participant Greg Cole, Director of High Performance Strategic Networking Initiatives at NCSA. "It was a historic first when we were invited to participate in this formerly-Russian-only conference last year. The necessary relationship of trust was initiated, and with Ronn's and Tony's participation this year, the Russians made a commitment to work with us on a broad measurement infrastructure. It is difficult to overstate the importance of this to our research goals and, in general, illustrating the robustness and capability of the joined U.S.-Russian network infrastructure to support science and engineering cooperation. Although it required two years of conversation, travel, and time commitment from Ronn, this is 'warp speed' for such major U.S.-Russia progress."

As a result of these efforts, an NLANR/MNA AMP device will be installed at the Russian central research network facility in Moscow.

NLANR/MNA managers Ritke, McGregor, and Jörg Micheel, manager of the PMA project, are continuing discussions on potential measurement activities and collaborations. Ritke, Cole, and Natasha Bulshova of NaukaNet met with Alexei Platonov, the conference organizer, who affirmed the Russian commitment to continuing the collaboration, with both the Nauka-Net program and during the planned five-year Russia/China/U.S. GLORIAD follow-on program next year. A one-year interim "Little GLORIAD" program of the three countries will serve as a test project for the subsequent major effort to set up a broad measurement infrastructure, to ensure the success of joint scientific networking endeavors of these three world powers.

Pacific Rim

TransPAC supports international collaborations in many fields of basic science, technology, engineering and medicine. Participating projects include the Grid Physics Network for distribution and analysis of experimental results in high-energy physics; the Asia-Pacific Bioinformatics Network, which provide genomic data, computational resources, and community support for bioinformatics; the Joint Program for Arctic Atmosphere Observation between geophysicists at the University of Alaska Fairbanks and the Japanese Communications Research Laboratory; and collaboration between Japanese and American astronomers in the Sloan Digital Sky Survey (SDSS).

TransPAC connects international research and education networks associated with the Asia Pacific Advanced Network (APAN) to the Internet2 Abilene network and to other global networks. To monitor performance of this vital trans-Pacific link, NLANR AMP machines are now hosted at the Tokyo APAN Network Operations Center and in Seattle, at both ends of the TransPAC link. In addition, Micheel has collaborated with John Hicks of Indiana University, the High Performance Computing and Communications applications engineer for TransPAC, to create passive measurement monitors for use on TransPAC network traffic.

NLANR/MNA is a founding member of PRAGMA, the Pacific Rim Applications and Grid Middleware Assembly, an open organization in which Pacific Rim institutions collaborate to develop grid-enabled applications and deploy the infrastructure throughout the Pacific Region to facilitate sharing of data, computing, and other resources. PRAGMA promotes visiting scholars' and engineers' programs, builds new collaborations, and assists trans-Pacific network deployment. A series of workshops was initiated in March, 2002 at SDSC, followed by workshops in Korea, Japan, and Australia); NLANR/MNA co-principal investigator Ritke has been a presenter at all but one of these meetings.

Korea

The Korea Research Environment Open NETwork (KREONET) is one of five national network systems in Korea, providing advanced Internet infrastructure and services to the science and technology communities, include researchers working for approximately 300 government-funded research and development centers, universities, and industrial R&D centers. KREONET is run by KISTI (the Korea Institute of Science and Technology Information), a key institution for national knowledge information infrastructure. KREONET is connected to STAR TAP on a 45Mbps link, providing high bandwidth to the Korean research and education community.

With the assistance of of NLANR/MNA's AMP staff, researchers Manhee Lee, Kisung Ryu, and Ok-Hwan Byeon of KREONET created a Korean AMP mesh consisting of 12 inter-communicating AMP monitors for KREONET-wide monitoring efforts. The data from the Korean AMP program is made available to network engineers, scientists, and students on the Web.

KREONET adopted the NLANR/MNA AMP monitor for its research efforts, and has been successfully monitoring 117 sites and assisting KREONET Network Operations Center members in analyzing the status of the network.

KREONET plans to collaborate with other institutes and to further expand the mesh, to use AMP monitors to measure network performance among other Asian countries. At a recent APAN meeting held in Thailand, many participants were interested in hosting AMP devices. If an AMP mesh is deployed throughout Asia, it will enable Asian network engineers to monitor the network status across Asia -- and it will further enhance Asian efforts at technical collaboration.

In addition, the researchers plan to use AMP to develop Grid applications. The KISTI supercomputing center is in charge of the Korean national Grid project. AMP devices can measure end-to-end performance among Grid application developers and assist in the development of their Grid applications. These efforts are proceeding in coordination with PRAGMA.

Taiwan

Researchers at the National Center for High-Performance Computing (NCHC) in Taipei have an active national monitoring effort. They have hosted two NLANR/MNA AMP monitors, at the island's northern and southern Network Operations Centers. The first of their own mesh of AMP monitors recently has been deployed, and measurements between Taiwan's AMP and the NLANR/MNA AMPs are under way. In addition, the NCHC researchers are interested in hosting an NLANR/MNA AMP as part of their 40-gigabit network upgrade, the Taiwan Advanced Research and Education Network (TWAREN) Initiative.

Thailand

Thailand currently hosts an NLANR AMP and PMA monitor. Network research in Thailand is coordinated by Uninet, a consortium of all the nation's public universities and several of its private ones. In this case, the collaboration will proceeds with the Thais buying and installing PC-based monitor hardware on the networks, and NLANR/MNA will remotely load the AMP operating software into the devices.

Australia

The Australian Academic and Research Network (AARNet) installed an AMP monitor supplied by NLANR in Sydney only four months after the HPIIS conference. Australian researchers and engineers quickly realized the need to develop a national measurement infrastructure to monitor the performance of links between AARNet, the advanced GrangeNet network covering the Eastern Australian States, and the CeNTIE network of Australia's Commonwealth Scientific and Industrial Research Organisation (CSIRO). With advice and assistance from NLANR/MNA, an Australian AMP mesh is being created, starting with seven Australian AMP machines that monitor Gigabit Ethernet links and measure the performance of links to international sites of particular interest to the Australian research and education communities. The Australian AMP locations include Sydney, Melbourne, Perth, Adelaide, Canberra, and Brisbane. Data from these AMP machines will be made available to the public.

Canada

The Canadian government has made a long-term commitment to build a national optical R&D Internet network, CA*net 3, that will operate at up to 40 Gigabits per second, built from the ground up to carry Internet traffic. The first NLANR/MNA AMP monitor was installed on CA*net 3 in November 2001. As a result of advice and assistance from NLANR/MNA, CANARIE, the Canadian network research organization, now has its own AMP meshes deployed on CA*net 3. Two NLANR/MNA AMP monitors currently are installed in this mesh; one is at the Advanced Research and Development Network Operations Center in Ottawa, and the other is in at the Reseau Interordinateurs Scientifique Quebecois (RISQ) in Montreal.