NLANR/MNA International Collaborations

NLANR/MNA Builds an International Infrastructure for Network Research and Troubleshooting

A San Diego Supercomputer Center White Paper

August 8, 2005

Executive Summary

The NSF-funded National Laboratory for Applied Network Research (NLANR) provides technical, engineering, and traffic analysis support for NSF High Performance Connections sites and specialized high-performance network service providers such as Internet2. The Measurement and Network Analysis team (NLANR/MNA), with headquarters at the San Diego Supercomputer Center (SDSC) – on the campus of the University of California, San Diego – conducts performance and flow measurements for HPC sites. The group has deployed a constellation of more than 160 active and passive traffic monitoring devices at network sites across the nation and in selected cooperating foreign countries.

International collaborative efforts began cautiously, but quickly expanded as foreign research organizations became aware of the benefits of collaborating with NLANR/MNA. The research infrastructure now extends to other continents, and network researchers and system administrators around the world are enthusiastically collaborating with the SDSC-based group – both hosting NLANR/MNA devices on their own networks and setting up measurement and analysis research projects of their own. NLANR/MNA participates in cooperative network measurement research efforts with groups from Australia, Brazil, Canada, Chile, China, Finland, Germany, Hungary, India, Ireland, Italy, Japan, Korea, Malaysia, Mexico, the Netherlands, New Zealand, Norway, Pakistan, Russia, Singapore, South Africa, Spain, Switzerland, Taiwan, Thailand, Tunisia, and the United Kingdom.

Performance measurement will increase in importance with the growth of traffic on international links and as multinational projects become more dependent on them. NLANR/MNA has been able to successfully initiate international cooperation in network measurement through a combination of personal and professional contacts, via participation in international organizations and conferences that promote computer networking, and as a result of the group´s reputation in the field, Web presence, and data archives that have prompted other research groups to make contact.

1.0 Introduction

The Measurement and Network Analysis group of the National Laboratory for Applied Network Research (NLANR/MNA) assesses the performance of next-generation computer networks – measuring the flow of message traffic, analyzing performance issues, and making all of the data, analyses and tools available to the community – so they can be tuned for maximum end-to-end performance. In particular, network measurements are essential for identifying and locating problems (malfunctions, bottlenecks, inefficiencies, incompatibilities, etc.) in ultrafast research networks and in high-speed international links.

“Although NLANR´s measurement and analysis efforts may not be visible to casual users of the Internet, all users ultimately benefit,’ said Tom Greene, former senior program director for the National Science Foundation´s Division of Advanced Networking Infrastructure and Research. “Measurement and analysis tools are essential for solving performance issues, and we all need solutions there.’

The NSF-funded NLANR/MNA group has created a network analysis infrastructure to enable their own research and to support the efforts of outside researchers, systems administrators, and students. The group has deployed a constellation of more than 160 active and passive traffic monitoring devices at network sites across the nation and in selected cooperating foreign countries. NLANR´s infrastructure also includes a growing collection of software tools and methods, raw data, and analyses and visualizations, available for use by other network researchers, engineers, and systems administrators.

Most of the monitoring devices have been located on high-performance networks operating in the United States. International expansion began cautiously four years ago; now the research infrastructure extends to other continents, and network researchers and system administrators around the world are enthusiastically collaborating with NLANR/MNA researchers, setting up measurement and analysis research projects of their own.

1.1 Active and Passive Data Collection

NLANR/MNA research has two main components. The Passive Measurement and Analysis (PMA) project uses information gleaned from observing network traffic, without interacting with the networks themselves. A low-technology analogy for this project would count envelopes, weigh them, and sort them by postmark to determine how efficiently the U.S Postal Service is handling the mail. The PMA project uses network message packet header traces instead of envelopes, but the principle is the same.

The Active Measurement Project (AMP) performs site-to-site active measurements and analyses, which enable network researchers and engineers to track problems and changes in network performance, by inserting test messages into the networks it studies and observing their progress through the systems. A Postal Service analog of the AMP project would send test letters between many locations, noting their travel times and postmarks.

More than 150 AMP monitors are deployed on high-speed research networks in the United States and in other countries. Monitors are “full mesh’ configurations (i.e., each monitor in a mesh measures to every other monitor in that mesh, with monitors shared between meshes. Measurements include message path, round-trip time, and packet loss. In addition to gathering data, the AMP project also analyzes long-term, large-scale trends in message flow patterns.

The monitoring devices for both projects are rack-mounted PCs with high-speed network cards, installed in the network equipment racks of universities and research institutions that participate in the two projects. All participation is voluntary, and both NLANR/MNA and participating sites are meticulous about maintaining the privacy of network users. To use another Postal Service analogy, the monitors count envelopes and examine postmarks and Zip codes, but they do not look at the full addresses or open any of the envelopes or read the letters.

“In addition to not looking at message content, we go to a lot of trouble to anonymize any address information before we ever see it,’ said Hans-Werner Braun, founder of NLANR and principal investigator of the Measurement and Network Analysis group. “It´s like encrypting the Zip codes with a key that we don´t have – it´s non-reversible, so we can´t tell what the addresses really were, but we use the anonymized versions of the addresses for statistical purposes.’

The AMP monitor is a low-profile, rack-mountable industrial PC that contains specialized high-speed network interface cards.

1.2 Using AMP Measurements for Troubleshooting

AMP monitors measure the “heartbeat’ of the high-performance connection networks, querying the networks over 15,000 pathways every minute. The current AMP architecture consists of two meshes, the HPC mesh with more than 140 sites and an international mesh that includes all international sites and a small subset of the HPC mesh. In each mesh, every monitor sends test messages to all the other monitors in the mesh. The measured variables currently include round trip time, packet loss, topology, and throughput (user/event driven).

The information gathered by the AMP devices is very useful to network engineers for troubleshooting. As an example, round trip time (RTT) measurements reveal increased latency between Sydney, Australia and Arizona State University as the result of a route change that was triggered by router maintenance.

This graph shows the round trip time of a test message, from the source AMP to the destination and back again. On the AMP website, clicking the [routes] link reveals the path from the source to the destination; from there, the reverse path can be viewed through the [reverse] link.

Route information is collected from traceroute commands executed approximately every 10 minutes. Route graphics reveal useful information, as shown by this visualization of the IP hop count over the route from the source to the destination. Anomalies stand out quite clearly. On the AMP website, by choosing a time (a position on the x-axis) and moving vertically up the graph, you cross each hop taken in the path at that time. The browser´s status line will show the DNS name for that hop. Clicking on the graph goes to a page with the traceroute information for that path and time.

Visualization of the IP hop count over the route between Sydney, Australia and Arizona State University.

Correlations with network events also reveal useful information. The AMP website provides event bar plots for this purpose. Colored horizontal bars below the route graphic (discussed above) indicate the durations of network events that we have been advised of from Abilene, vBNS, or StarTap, or in some cases our own activities. The event bar time frame matches that of the route graphic immediately above it. On the AMP website, clicking on a bar will take you to the text of the notification. The corresponding event bar plot. Bars are color-coded to indicate the sources of events; the green event bar shown here indicates that the event source was the Abilene Network Operations Center.

2.0 NLANR Involvement in International Network Research

In 1998, Tony McGregor, professor of computer science at Waikato University in New Zealand, spent a six-month sabbatical at SDSC, which led to his joining NLANR as a manager. Since then, researchers at Waikato have been an important element of the NLANR/MNA research team.

In 1999, researchers in Norway, New Zealand, and Israel borrowed single network monitor units from NLANR/MNA on an experimental basis. International interest in cooperative network research was low-key, until one small event started a landslide.

2.1 HPIIS Spurs Interest

The NSF created the High Performance International Internet Services (HPIIS) Program in 1997 to provide international researchers access to high-performance networks in the United States and to give U.S. educators access to researchers and international high-performance networks around the globe. HPIIS project participants include AMPATH (which links most of the countries in Central and South America to the U.S.), Euro-Link (between the U.S. and Europe), the TransPAC link (between the U.S. and Japan), NaukaNet (formerly called MIRnet, between the U.S. and Russia), and NSF´s STAR TAP and StarLight international access points.

As a result of hosting an NSF-funded HPIIS workshop at SDSC in 2001, NLANR/MNA is now conducting joint research with all four HPIIS grant projects. The SDSC researchers also are collaborating with three additional countries not in the HPIIS program that sent representatives to the workshop. As a result of requests for collaboration from interested foreign research organizations, new AMP monitors were placed in Australia, Canada, and Korea shortly after the HPIIS conference.

Thirty researchers, representing organizations that included about two dozen different countries, met for face-to-face discussions of issues. Suddenly everyone saw the advantages of international cooperation and coordinated research. The enthusiasm was contagious.

2.2 International Cooperation

NLANR/MNA is and is perceived as a neutral intermediary, which facilitated cooperation and trust across many different organizational and national boundaries.

The NLANR/MNA group now has AMP machines up and running in Australia, Brazil, Canada, Chile, China, Finland, Germany, Hungary, Ireland, Japan, Korea, Mexico, the Netherlands, New Zealand, Norway, Pakistan, Singapore, Spain, Switzerland, Taiwan, Thailand, and the United Kingdom; a machine is on site in Russia, but the software is not yet initialized (ready to collect data). South Africa, Tunisia, India, and Malaysia, soon will host monitors as well. We are in discussion with groups from Algeria, Egypt, Italy, Kenya and Senegal. The StarLight international links include AMP monitors, and many measurement machines are located at individual U.S. sites that serve as international entry/exit points on high-speed networks. And as part of the cooperative effort, all of these projects are sharing their data with other researchers around the world.

The result is that NLANR/MNA is well on the way to establishing a world-wide infrastructure for gathering performance data from high-speed next-generation research networks, vitally important in an age of global communications.

But of equal importance is the intangible network of person-to-person contacts among scientists and engineers in dozens of countries that have been engendered by the program. Researchers and administrators are engaging in joint efforts, exchanging information, and helping to solve one another´s problems using a range of different collaboration models, all started as experimental efforts facilitated by NLANR/MNA´s activities.

NLANR/MNA has participated in cooperative network measurement research efforts with Australia, Brazil, Canada, Chile, China, Finland, Germany, Hungary, India, Ireland, Italy, Japan, Korea, Mexico, Malaysia, the Netherlands, New Zealand, Norway, Pakistan, Russia, Singapore, South Africa, Spain, Switzerland, Taiwan, Thailand, Tunisia, and the United Kingdom.

2.3 Judging Success

The success of the effort can be judged by another workshop that NLANR/MNA hosted at SDSC less than two years after the HPIIS meeting. PAM2003, the international passive and active network measurement conference, was a three-day event attended by more than 80 researchers and students from Australia, Belgium, Canada, Croatia, France, Germany, Italy, Japan, the Netherlands, Norway, the United Kingdom, and Sweden as well as from the United States. A record number of papers was submitted to the workshop – of 92 submissions, 20 were accepted as full papers and five were accepted as posters.

Technology transfer is an important aspect of the collaborations. AMP technology is now being used in parallel national programs to instrument and characterize the performance of the domestic research networks of Australia, Korea, and Taiwan; research groups in Thailand, Brazil, Russia, and China have expressed interest in building similar national infrastructures.

And the transfer flows in more than one direction – these cooperating national efforts are being integrated into the infrastructure established and run by NLANR/MNA, with new ideas shared among all participants. For example, presentation topics by international participants at PAM2003 included leading-edge research in active measurement (sending test packets and observing their progress through the network), passive measurement (observing actual traffic on “live’ networks), performance metrics (developing indicators to characterize traffic behavior), traffic statistics (to develop models of “real’ network traffic), and visualization (finding effective ways to display what is happening in a network).

Another gauge of the usefulness of the international infrastructure effort and the importance of person-to-person contacts is our success in dealing with the issue of national information security. Many of the nations involved in collaborations with NLANR/MNA have traditionally been quite sensitive about cybersecurity. Placing measurement devices developed in the United States on other countries´ national data networks that facilitate state-of-the-art scientific and technological research was no mean feat. The buildup of trust between researchers and governmental organizations has been absolutely essential to the success of our efforts. The project´s openness, non-competitive nature, and general attitude of sharing data for mutual benefit have allayed suspicions and overcome institutional barriers, with the result that cooperative efforts are proceeding with surprising smoothness.

“The continuous growth of the NLANR Measurement and Network Analysis effort has fostered the growth of wider understanding of the importance of network measurement and the international demand for collaborations with the NLANR team,’ said Tom Greene, former senior program director for the NSF´s Advanced Networking Infrastructure and Research division.

NLANR International AMP Mesh, as of August 8, 2005

3.0 Specific Projects

NLANR/MNA´s encouragement of international networking research collaborations has caused an avalanche of activity in the field. Initial gestures of support have fostered more support, and more substantive support – as more international collaborations become active, there is more data upon which to do research, there is more cross-fertilization among researchers, and the benefits of collaboration become more apparent to the researchers and to their funding agencies.

With the assistance of Internet2´s Ana Preston and Heather Boyles, Ritke made presentations at several Internet2 Member Meeting International Focus sessions. This led to further contacts with groups in Brazil and Mexico that were interested in active measurement (see below).

3.1 NSRC

NLANR/MNA is working with Steve Huter from UO-NSRC. Based at the University of Oregon, the Network Startup Resource Center (NSRC) project provides technical information, engineering assistance, training, and equipment to universities, research institutions and networking organizations in developing areas.

The NSRC's primary goal is to make it easier for U.S. scientists and engineers to collaborate via the Internet with their international colleagues. NSRC is partially supported by the National Science Foundation

The roots of the NSRC trace back to a volunteer effort to support networking in southern Africa in 1988, when Randy Bush designed and helped deploy a multi-country (South Africa, Botswana, Namibia, Zimbabwe, and later many others) network using various technologies (FidoNet, UUCP, and TCP/IP). The NSRC was formalized by Randy Bush and Dr. John Klensin with a grant from the National Science Foundation in 1992, the first of several NSF grants to support NSRC activities. Based at the University of Oregon Computing Center since 1996, the NSRC project has assisted universities, research institutes, Internet Service Providers, non-governmental organizations, governmental and supranational agencies, industry, and numerous international networking organizations. For more information see: http://www.nsrc.org/.

3.2 AMPATH and WHREN-LILA

The AmericasPath project, AMPATH, began as a collaboration between Florida International University (FIU) and Global Crossing, and is now supported in part by grants from the NSF. Using Global Crossing´s land-based and undersea optical fiber network, AMPATH interconnects the research and education networks in South and Central America, the Caribbean, and Mexico to U.S. and foreign networks via the high-speed Abilene network backbone of the Internet2 consortium. Participants also utilize the StarLight connection points.

The NSF is funding FIU´s proposal for the Western Hemisphere Research and Education Network and Links Interconnecting Latin America (WHREN-LILA) to create an all encompassing network for the western hemisphere, interconnecting established international peering exchanges in North America (Miami, Los Angeles, Seattle, Chicago and New York) with emerging international peering exchanges in Latin America (Sao Paulo, Santiago and Tijuana).

NLANR engineers worked with Eric Johnson, the network engineer for AMPATH, and with Julio Ibarra and Heidi Alvarez of FIU to install AMP and PMA monitors in the network access point in Florida. The active monitor is providing data, and the passive monitor is sending traffic traces that will be made publicly available.

AMPATH has implemented a connection to the Academic Network at São Paulo (ANSP), the network of the State of São Paulo in Brazil. A research group in Brazil installed an AMP monitor on the link from the AMPATH connection in Florida to Brazil in 2002. This AMP is being moved to the new WHREN-LILA International Research Network Connection (IRNC) in Brazil.

3.3 CUDI

An AMP monitor is in place at the Network Operations Center of Corporacion Universitaria para el Desarollo de Internet (CUDI) in Mexico City, the Mexican education and research network. Created in April 1999, CUDI is organized as a non-profit corporation of universities and research centers. There is a 155 Mbps ATM-based link between Tijuana and San Diego. Our activities have been facilitated by Ana Preston of Internet2.

3.4 TransPAC

TransPAC supports international collaborations in many fields of basic science, technology, engineering and medicine. Participating projects include the Grid Physics Network for distribution and analysis of experimental results in high-energy physics; the Asia-Pacific Bioinformatics Network, which provide genomic data, computational resources, and community support for bioinformatics; the Joint Program for Arctic Atmosphere Observation between geophysicists at the University of Alaska Fairbanks and the Japanese Communications Research Laboratory; and collaboration between Japanese and American astronomers in the Sloan Digital Sky Survey (SDSS). NLANR/MNA has AMPs at two of the new IRNC TransPAC2 links nodes (Japan and China). Our activities in TransPAC have been assisted by Jim Williams.

3.5 PRAGMA

NLANR/MNA is a founding member of PRAGMA, the Pacific Rim Applications and Grid Middleware Assembly, an open organization in which Pacific Rim institutions collaborate to develop grid-enabled applications and deploy the infrastructure throughout the Pacific Region to facilitate sharing of data, computing, and other resources. PRAGMA promotes visiting scholars´ and engineers´ programs, builds new collaborations, and assists trans-Pacific network deployment. A series of workshops was initiated in March, 2002 at SDSC, followed by workshops in Korea, Japan, Australia, and Taiwan); NLANR/MNA co-principal investigator Ritke has been a presenter at three of these meetings. NLANR/MNA´s activities in PRAGMA have been greatly assisted by Peter Arzberger and Teri Simas of UC San Diego.

Most of the PRAGMA testbed sites are covered by NLANR AMPs. The only testbed sites not currently covered are India, Malaysia, and Singapore, and we have discussed hosting an AMP device with representatives from all three sites. Researchers at the University of Hyderabad in India have just submitted the request form for an AMP.

3.6 NaukaNet

RELARN is a Russian voluntary association of scientific and educational organizations, initiated in 1992 by the Ministry of Science, the Russian Academy of Sciences, and the Kurchatov Institute research center. RELARN supports non-commercial information exchange using computer networks technologies, coordinates projects on computer networks for science and education development, promotes interaction with research computer networks in Russia and abroad, and facilitates scientific contacts by organizing conferences, competitions, and exhibitions.

NaukaNet delivers enhanced international network services to Russia´s emerging high-performance networks and gives them access to American researchers by connecting optical networks in the Moscow and St. Petersburg areas to the U.S. network infrastructure via STAR TAP. It links numerous collaborative projects in fields ranging from high-energy physics and the control of nuclear materials to geological studies of the earth´s crust and environmental engineering. The NaukaNet initiative began with HPIIS support to the University of Tennessee, Knoxville, with matching support from the Ministry for Science and Technology of the Russian Federation.

NLANR/MNA co-principal investigator Ritke attended and gave presentations on measurement activities to RELARN meetings held in Russia in early June 2002 and June 2003. Tony McGregor, manager of the AMP project, also attended and presented at the June 2003 RELARN meeting.

“I don´t think we could have brought a more representative delegation [of] the U.S. communities in science, education, grid development and networking,’ said fellow participant Greg Cole, Director of High Performance Strategic Networking Initiatives at NCSA. “It was a historic first when we were invited to participate in this formerly Russian-only conference last year. The necessary relationship of trust was initiated, and with Ronn Ritke´s and Tony McGregor´s participation this year, the Russians made a commitment to work with us on a broad measurement infrastructure. It is difficult to overstate the importance of this to our research goals and, in general, illustrating the robustness and capability of the joined U.S.-Russian network infrastructure to support science and engineering cooperation. Although it required two years of conversation, travel, and time commitment from Ronn, this is ‘warp speed´ for such major U.S.-Russia progress.’

As a result of these efforts, an NLANR/MNA AMP device will be installed at the Russian central research network facility in Moscow. Independent of this initiative, the NLANR/MNA team also is currently taking measurements to more than 30 cooperating sites in Russia from the NLANR AMP device on the NaukaNet link at the StarLight facility in the United States.

NLANR/MNA managers Ritke, McGregor, and Joerg Micheel, manager of the PMA project, are continuing discussions on potential measurement activities and collaborations. Ritke, Cole, and Natasha Bulshova of NaukaNet met with Alexei Platonov, the conference organizer, who affirmed the Russian commitment to continuing the collaboration, with both the NaukaNet program and during the five-year Russia/China/U.S. GLORIAD program.

3.7 GLORIAD

Russia and China are connecting their science networks, and will join with the United States to complete the Global Ring Network for Advanced Application Development (GLORIAD) – a ring around the northern hemisphere for joint scientific and educational projects. When completed, GLORIAD will be a 10 Gbps lightwave network, based on a hybrid circuit-switched and routed architecture, around the entire northern hemisphere to support science and education cooperation. The network will run through the StarLight facility in Chicago (with partners at the University of Illinois at Chicago and Northwestern University). It will cross the Atlantic ocean to the NetherLight facility in Amsterdam, and will continue from there to Moscow, to the Russian science city of Novosibirsk, across Siberia to the Chinese border, and to Beijing, to Hong Kong, and then across the Pacific ocean to complete the ring in Chicago.

Collaborative efforts began with the one-year “Little GLORIAD’ test project, and continue with the full GLORIAD effort. NLANR/MNA and the GLORIAD team will develop a network performance measurement infrastructure to gauge network performance between end-point sites in the U.S., Russia, and China. The NLANR group will deploy active measurement devices on networks on the StarLight international access point and on networks in Russia and China.

An AMP monitor already has been deployed at the Computer Network Information Center (CNIC) in Beijing to support GLORIAD. An AMP has been delivered to Russia, but it is not yet operational. The Hong Kong Exchange Point, where GLORIAD and TransPAC2 will connect, is hosting an AMP measurement device; this is the only open exchange point for the Internet in Asia. Some of the other GLORIAD partners – Korea/KISTI, Netherlands/SURFnet, Canada/CANARIE already utilize AMPs.

3.8 StarLight/STAR TAP

Euro-Link is an NSF-funded initiative to connect European and Israeli national research networks to American networks, supported through the HPIIS program as a next-generation Internet initiative for international research collaboration. Euro-Link consortium members NORDUnet, SURFnet, RENATER2 and CERN, along with partner HPIIS network consortia TransPAC and MIRnet, connect to the U.S. through the Science, Technology, and Research Transit Access Point (STAR TAP).

StarLight, the optical STAR TAP initiative, is an advanced infrastructure and proving ground for services on high-speed (gigabit and 10-gigabit) optical networks. StarLight is run by the Electronic Visualization Laboratory at the University of Illinois at Chicago, the International Center for Advanced Internet Research (iCAIR) at Northwestern University, and the Mathematics and Computer Science Division at Argonne National Laboratory, in partnership with Canada´s CANARIE and Holland´s SURFnet programs.

StarLight and STAR TAP are the world´s most strategic international access points, with an enormous volume of international traffic. An NLANR/MNA AMP monitor is located at the SBC GigaPop in Chicago on the StarLight HPC connection. A PMA device is located at the STAR TAP GigaPop and takes passive measurements on the HPC link; passive optical taps already have been pre-installed at StarLight to avoid having to take down the operational link when NLANR/MNA´s PMA monitors are installed and deployed in the near future. Our efforts have been facilitated and encouraged by Tom DeFanti of STAR TAP/StarLight.

3.9 International Research Network Connection (IRNC) Awards

As follow-on to High-Performance International the Internet Services (HPIIS) awards of 1997, in early 2005 the NSF has made a new set of awards to for network connections linking U.S. research networks with peer networks in other parts of the world that provide economy of scale and demonstrate the ability to link communities of interest with broad services. These awards include the US-Russia-China GLORIAD project and the TransPAC2 project to serve the Pacific Rim, the TransLight/StarLight project to serve Europe, the WHREN-LILA project to serve the Americas and the TransLight Pacific Wave Project with connections to Australia and the West coast of the US.

TransPAC2 maintains U.S. connectivity with Japan, in the form of a 10-gigabit connection between Los Angeles (PacificWave) and Tokyo, with connectivity to other parts of Asia through the Hong Kong Exchange Point.

The TransLight/StarLight project provides two 10 gigabit links between the U.S. and Europe, from MAN LAN in New York to the GEANT hub in Amsterdam.

The WHREN-LILA project funds U.S. connectivity to RedCLARA through a 1.2 gigabit connection between Miami and Sao Paulo, Brazil (evolving to 2.5 gigabits) and a 1 gigabit connection between San Diego and Tijuana, Mexico.

4.0 Coordination with Foreign Network Research Projects

Coordination between NLANR/MNA and its international counterparts has been established through direct and indirect means. Some contacts were initiated by direct contacts between NLANR personnel and the staff of foreign network research projects. Other cooperative endeavors resulted from involvement of NLANR/MNA and its foreign counterparts in “third-party’ research organizations.

4.1 Australia

The Australian Academic and Research Network (AARNet) installed an AMP monitor supplied by NLANR in Sydney only four months after the HPIIS conference. Australian researchers and engineers quickly realized the need to develop a national measurement infrastructure to monitor the performance of links between AARNet, the advanced GrangeNet network covering the Eastern Australian States, and the CeNTIE network of Australia´s Commonwealth Scientific and Industrial Research Organisation (CSIRO). These organizations´ cooperation with NLANR/MNA was facilitated by Bruce Morgan and George McLaughlin of AARnet.

With advice and assistance from NLANR/MNA, an Australian AMP mesh is being created to monitor local research network links and to measure the performance of links to international sites of particular interest to the Australian research and education communities. The Australian AMP sites include Sydney, Melbourne, Perth, Adelaide, Canberra, and Brisbane (see http://www.aarnet.edu.au/). The AMP units have been deployed but not activated; these local AMP devices soon will be moved and new IP addresses will be assigned to take advantage of new upgrades to the Australian networks.

4.2 Brazil

The RNP group in Brazil has deployed three of their own AMP machines locally to evaluate the feasibility of creating a national active measurement project. This has served as a valuable beta test site for NLANR/MNA´s Tony McGregor in the development of the new AMP software package. The NLANR AMP will be moved from the old AMPATH connection to the new WHEN-LILA International Research Network Connection (IRNC) in Brazil.

4.3 Canada

The Canadian government has made a long-term commitment to build a national optical R&D Internet network, CA*net 3, that will operate at up to 40 Gigabits per second, built from the ground up to carry Internet traffic. Cooperation with NLANR/MNA was facilitated by Herve Guy of CANARIE.

The first NLANR/MNA AMP monitor was installed on CA*net 3 in November 2001. As a result of advice and assistance from NLANR/MNA, CANARIE, the Canadian network research organization, now has its own AMP meshes deployed on CA*net 3. Two Canadian active monitors currently are installed in this mesh; one is at the Advanced Research and Development Network Operations Center in Ottawa, and the other is in at the Reseau Interordinateurs Scientifique Quebecois (RISQ) in Montreal.

4.4 Chile

The University of Chile in Santiago just began hosting an operational AMP monitor. In addition, Chile soon will be a PRAGMA testbed site.

4.5 China

NLANR´s involvement in the PRAGMA and GLORIAD efforts has involved several years of contact with Chinese researchers, including visits by Ronn Ritke of NLANR and Peter Arzberger of PRAGMA and visits by Director Baoping Yan of the Computer Network Information Center (CNIC) to SDSC. Cooperation with NLANR/MNA is being facilitated by Director Yan and Nan Kai of CNIC.

NLANR has been quite successful in gaining entry to Chinese networks and coordinating with Chinese researchers. An AMP unit has been deployed in Beijing, China at the Computer Network Information Center (CNIC). Another AMP is located at the Hong Kong Exchange Point, where the NSF-funded GLORIAD and TransPAC2 international networks will connect. NLANR/MNA has been invited to deploy seven additional AMPs at the GigaPoPs in Beijing, Shanghai, Guangzhou, Shenyang, Changchun, Chengdu, and the CNC/CSTNET Network Management Center as part of China´s Next Generation Internet.

4.6 Finland

Finland currently hosts an operational AMP monitor at the Helsinki University of Technology.

4.7 Germany

Germany hosts an operational AMP monitor at Leipzig University.

4.8 Hungary

Hungary hosts an operational AMP monitor at Eotvos University in Budapest.

4.9 India

India is a PRAGMA testbed site. Researchers at the University of Hyderabad in India are hosting an AMP device.

4.10 Ireland

Ireland currently hosts an operational AMP monitor at a HEANet POP, located in the Brooklawn House at the corner of Crampton Avenue and Shelbourne Road, in Dublin.

4.11 Italy

Italian researchers have expressed interest in hosting an AMP monitor.

4.12 Japan

TransPAC connects international research and education networks associated with the Asia Pacific Advanced Network (APAN) to the Internet2 Abilene network and to other global networks. To monitor performance of this vital trans-Pacific link, an AMP monitor is now hosted at the Tokyo APAN Network Operations Center; one soon will be placed in Los Angeles, to cover both ends of the TransPAC link. Cooperation between APAN and NLANR/MNA was facilitated by TransPAC, in particular through the activities of Yasuichi Kitamura, the NOC Team Measurement Chair.

In addition, Micheel has collaborated with John Hicks of Indiana University, the High Performance Computing and Communications applications engineer for TransPAC, to create passive measurement monitors for use on TransPAC network traffic. Japan is participating in TransPAC2.

4.13 Korea

The Korea Research Environment Open NETwork (KREONET) is one of five national network systems in Korea, providing advanced Internet infrastructure and services to the science and technology communities, include researchers working for approximately 300 government-funded research and development centers, universities, and industrial R&D centers. KREONET is run by the Korea Institute of Science and Technology Information (KISTI), a key institution for national knowledge information infrastructure. KREONET is connected to STAR TAP on a 45Mbps link, providing high bandwidth to the Korean research and education community.

With the assistance of NLANR/MNA staff, researchers Manhee Lee, Kisung Ryu, and Ok-Hwan Byeon of KREONET created the AMP KREONET project, a local mesh based at the KISTI Supercomputing Center (see http://amp.kreonet2.net/). This is a Korean local mesh consisting of 16 inter-communicating monitors for KREONET-wide monitoring efforts. The data from the Korean AMP program is made available to network engineers, scientists, and students via the Web. KREONET plans to collaborate with other institutes and to further expand the mesh, using AMP monitors to measure network performance among other Asian countries.

In addition, the researchers plan to use AMP to develop Grid applications. The KISTI supercomputing center is in charge of the Korean national Grid project, and AMP devices will measure end-to-end performance for Grid application developers and assist in the development of Grid applications. These efforts are proceeding in coordination with PRAGMA. Korea also is participating in the GLORIAD project.

4.14 Malaysia

The University of Sains, Malaysia is a PRAGMA testbed site and is hosting an NLANR AMP device.

4.15 Mexico

Mexico hosts an NLANR AMP at the CUDI NOC in Mexico City and at a PRAGMA testbed site in Ensenada.

4.16 The Netherlands

The Netherlands is a GLORIAD participant, and hosts an AMP measuring device at the GEANT hub in Amsterdam.

4.17 New Zealand

New Zealand is developing their own version of a local AMP mesh. Coordination by Tony McGregor of NLANR/MNA, who is based at the University of Waikato, is greatly facilitating this effort. A total of 33 AMP devices have been deployed – 26 on a wireless networking project (see http://amp.crc.net.nz/src.php) and seven more on commercial network links at various sites in New Zealand (see http://erg.wand.net.nz/amp/). The AMP code is also being used by an ISP for internal testing. In addition, McGregor is conducting beta testing of the next-generation AMP software package in New Zealand.

4.18 Norway

Norway hosts an AMP measuring device at the Norwegian University of Science and Technology in Trondheim.

4.19 Pakistan

The University of Punjab is hosting a NLANR AMP machine.

4.20 Russia

We currently are performing one-way active measurements to more than 30 sites in Russia (see http://amp.nlanr.net/active/amp-naukanetnwu/russia/body.html). As part of the GLORIAD effort, we have shipped an AMP machine to Moscow; this machine is awaiting installation of AMP software.

4.21 Singapore

Nanyang Technical University, Singapore is hosting an AMP, is a PRAGMA testbed site, and hosted the PRAGMA 9 meeting.

4.22 South Africa

Two AMPs will be hosted in South Africa, one at the University of Cape Town and one at the TENET network POP.

4.23 Spain

Spanish researchers at the University of Catalunya, Catalunya are hosting a NLANR AMP device.

4.24 Switzerland

Kim Baldridges group at the University of Zurich is hosting an AMP monitor.

4.25 Taiwan

Researchers at the National Center for High-Performance Computing (NCHC) in Taipei have an active national monitoring effort. Cooperation between NCHC and NLANR/MNA was facilitated by both institutions´ membership in PRAGMA and by the efforts of Whey-Fone Tsai and Fay Sheu of NCHC.

The Taiwanese researchers host an NLANR/MNA AMP monitor at the island´s northern Network Operations Center and have deployed 3 of their own local AMPs. Taiwan is developing a domestic active measurement infrastructure. Che-nan Yang and NCHC researchers plan to expand the local AMP mesh to 11 machines as part of their 20-gigabit network upgrade, the Taiwan Advanced Research and Education Network (TWAREN) Initiative.

4.26 Thailand

Network research in Thailand is coordinated by UniNet, a consortium of the nation´s public and private institutions. Cooperation with NLANR/MNA was promoted by Wanchai Rivepiboon of the Advisory Committee to the Minister of Science and Technology.

Thailand currently hosts an NLANR AMP monitor. Future collaboration will consist of UniNet buying and installing PC-based monitor hardware at individual universities, with NLANR/MNA loading the AMP operating software into the devices and remotely operating the AMP units for UniNet to create a national measurement mesh. UniNet will host AMP software on their machines; they plan to use the new AMP software suite in establishing a local mesh. This effort will start with a single machine and expand from there. Thailand also is a PRAGMA testbed site (see http://www.uni.net.th/en/).

4.27 Tunisia

The University of Tunis is hosting an AMP.

4.28 The United Kingdom

The U.K. hosts an AMP measurement device at Cambridge University.

5.0 Collaboration Models

International collaboration between foreign researchers and the NLANR/MNA team has proceeded under several different models.

5.1 Elementary Collaboration

The simplest form of collaboration, and the least difficult to initiate, is the case in which a research group or an network access point in a foreign country agrees to host an NLANR/MNA active or passive monitor on their network; information collected from the device is made available to any researchers who care to access it. One of the first expansions of NLANR´s active measurement efforts beyond the United States occurred in 1999, when Norwegian graduate student Brynjar Viken returned to the Norwegian University of Science and Technology (NTNU) in Trondheim with an NLANR AMP unit after conducting research with NLANR´s Hans-Werner Braun at SDSC for six months.

5.2 Evolved Collaboration

The second model is a stage of collaboration that has gone one step beyond the passive collaboration stage. After hosting an NLANR monitor on their network, a foreign research group establishes an independent national network monitoring infrastructure based on the NLANR model, and shares the collected data with us (and with the rest of the world). The Korean and Canadian traffic analysis efforts are examples of this collaborative model, in which an advanced national effort evolved from placement of a single AMP monitor at a strategic location.

5.3 Assisted Collaboration

In this model, a foreign research group purchases and installs the PC hardware necessary to build one or more AMP or PMA monitors. The NLANR/MNA group downloads the software suite into this hardware. Once the devices are operating, NLANR and the hosting country share information with each other. If necessary, NLANR runs the data collection system remotely. Two examples of this model are Australia and Thailand.

5.4 Cooperative Activities with Ongoing Efforts

NLANR/MNA supports and participates in several international organizations that involve high-performance networking, including PRAGMA, NaukaNet, and GLORIAD. NLANR also has a long history of cooperation with StarLight and AMPATH. The MNA group is working with the new IRNC awards (GLORIAD, TransPAC2, and WHREN-LILA).

6.0 Placement Strategies

NLANR/MNA uses several different strategies in siting its active and passive monitor devices on foreign and international networks.

6.1 International Connection Site

One strategic site for NLANR/MNA monitoring equipment is at an international connection site in the United States through which a large volume of traffic flows. Two examples of this are the StarLight connection point in Chicago, and the AMPATH connection point in Florida.

6.2 Strategic Point on a Foreign Network

Another placement strategy embeds an AMP monitor at a strategic point on a foreign network where it can measure traffic between the U.S. and the host country, and can measure links from the host country to other countries. Japan is a good example of this siting strategy. The AMP at the APAN Network Operations Center in Tokyo provides link performance data for the TransPac link between the U.S. and Japan, and links between APAN Tokyo and various APAN countries – including a link from Seoul, Korea to the U.S. that runs through APAN that is not on the direct link between Daejon, Korea and StarLight in the U.S.

6.3 Strategic Point on Foreign Network to Monitor Three Patterns

A third approach is to site an AMP device at a strategic point on a foreign network where it can measure traffic on all three routing patterns – between the U.S. and the host country, over links within the host country on the host´s national network(s), and links between the host country to other countries. Korea exemplifies this approach, in which measurements are made for traffic between the host site (Daejon) to US (via the Kreonet2 international link), the host site to other sites on the host´s national net(s) (Kreonet), and between Korea country to other countries (see http://amp.nlanr.net/active/amp-korea/international/body.html). Australia is another; one well-placed AMP in Sydney collects data on national network links, on the link between Australia and the U.S., and between Australia and other countries (see http://watt.nlanr.net/active/amp-aarn/HPC/body.html).

7.0 Acknowledgements

We are indebted to Peter Arzberger, Hans-Werner Braun, Bill Chang, Greg Cole, Chip Cox, Doug Gatchell, Tom DeFanti, Tom Greene, Steve Huter, and many others for their assistance and encouragement.

This NLANR project, located at the University of California, San Diego and its San Diego Supercomputer Center is based on work sponsored by the National Science Foundation under cooperative agreement nos. ANI-0129677 (2002) and ANI-9807479 (1998). The Government has certain rights to the material presented here. Any use of this information or information from the NLANR/MNA website for research or commercial purposes must credit the National Laboratory for Applied Network Research and these cooperative agreements.

8.0 References

AARNet (Australia´s Research and Education Network) http://www.aarnet.edu.au/

AmericasPATH Network (AMPATH) http://www.ampath.fiu.edu/

Asia-Pacific Advanced Network (APAN) http://www.apan.net/

CANARIE http://www.canarie.ca/about/

Chinese Academy of Sciences http://english.cas.ac.cn/Eng2003/

Corporacion Universitaria para el Desarollo de Internet (CUDI) http://www.cudi.edu.mx/

Euro-Link http://www.euro-link.org/

GLORIAD http://www.gloriad.org/

Internet2 International Partnerships http://international.internet2.edu/

Korea Institute of Science and Technology Information (KISTI) http://www.kisti.re.kr/kisti/english/index_english.jsp

Korea Research Environment Open NETwork (KREONET) http://www.kreonet2.net/