Researchers based in Western Australia can no longer directly access nationally funded resources on the Nectar Research Cloud. In an email sent to researchers in WA, Nectar announced that this decision was taken because the Pawsey Supercomputing Centre, originally providing support to Nectar in WA, had “decided to discontinue providing resources to the Nectar Federation”.
Although the Nectar Research Cloud is nationally funded as part of the National Collaborative Research Infrastructure Strategy (NCRIS), Nectar has decided to close the door to WA researchers because of “the implications of the absence of a WA-based resource provider and co-investment from WA research institutions”.
This move has come as a surprise to institutions in WA, who as far as I have been able to tell, did not know that they were about to be cut off.
The decision raises critical questions about how an entire state of Australia can be cut off from nationally funded research infrastructure?
What is the Nectar Research Cloud?
Nectar (National eResearch Collaboration Tools and Resources) was established in 2009 with funding from the National Collaborative Research Infrastructure Strategy (NCRIS), which has invested A$2.8 billion since 2004 in research infrastructure. The original funding for Nectar was A$47 million from the Super Science Initiative and A$9.4 million under the NCRIS strategy 2013 program.
In total, Nectar has received A$61 million in government funding matched by co-investment of A$54 million from Australian universities and research organisations.
The remit of the organisation was to create eResearch tools, virtual laboratories, a research cloud and a secure and robust hosting service. The research cloud part is essentially a service similar to the basic functionality of cloud providers like Amazon Web Services, Google Cloud, Microsoft Azure and others.
Researchers could apply for access to computing resources in the cloud at no direct cost to them on a merit basis.
Some uses of Nectar resources have resulted in what are called “Virtual Laboratories”. These virtual laboratories host data and tools from disciplines such as genomics, astronomy, marine science and climate research.
Why not use commercial resources?
The big question regarding governments funding these types of infrastructure projects is why they would spend so much money when a range of commercial companies do the job already and one could argue, do the job better?
One immediate issue with funding computer infrastructure is that it only has a limited lifetime of 3 to 4 years before it needs completely replacing. Whilst this may be acceptable for extremely specialised super computers that are essential to certain types of research, the advantages for regular server computing resources is not so clear.
The field of cloud computing is evolving at an extremely rapid pace. Amazon is adding services to its cloud offerings on a monthly basis. It provides around 120 different cloud services that can be paid for on an hourly, monthly or yearly basis.
In contrast, the Nectar equivalent functionality as part of its computer cloud operation only offers the basic services of a virtual machine and storage.
Whereas commercial cloud providers are investing billions of dollars into new services, especially in the machine learning area, government funded infrastructure of this sort is simply not able to compete.
What next?
It is not clear how an organisation that has been funded to provide a national service can subsequently alter the conditions of service and resource provision to Western Australia. Nectar is in the portfolio of the Federal Government’s Department of Education and Training and so ultimately, it would be their responsibility to ensure that resources funded nationally are available to all researchers in Australia.
The suggestion by Nectar for researchers in WA was to ask another researcher in another state to apply for resources, negotiate or pay for access to a facility in another state, or “consider using a commercial cloud provider” and again pay for that service.
If that is indeed the case, it would be good if WA could get back its share of the A$61 million to spend on these options.
In preparing this post, I reached out to both the Pawsey Supercomputing Centre and Nectar for comment but had not received a reply by the time of posting. I will update if and when I receive a reply.
Update: 7th September 2017 12:20 AWST
In response to the article, the statement below was provided from Associate Professor Glenn Moloney Director, Nectar.
Nectar is providing a response to a number of points made in the article published in The Conversation on 5 September: Western Australian researchers get cut off from nationally funded research cloud; published in response to a communication from Nectar on 4 September 2017. A number of claims in the article are not supportable on the basis of that communication.
The communication from Nectar to Research Cloud users is available at: https://nectar.org.au/changes-access-nectar-research-cloud-western-australian-researchers
Nectar has also posted an additional statement clarifying the arrangements for support of the Research Cloud and the recent changes to arrangements in Western Australia: https://nectar.org.au/nectar-support-arrangements
Nectar continues to support access to the Research Cloud for Western Australian researchers
The author claims that “Nectar has decided to close the door to WA researchers” and that “It is not clear how an organisation that has been funded to provide a national service can unilaterally decide not to provide Western Australia with those resources.” [Note from editor: this sentence has now been revised by the author to the following: “It is not clear how an organisation that has been funded to provide a national service can subsequently alter the conditions of service and resource provision to Western Australia.”]
It is clear from the communication to Research Cloud users that Nectar has made no such unilateral determination. Rather, Nectar and our operating partners are considering the implications of the withdrawal by Pawsey of the Research Cloud capital infrastructure and associated operating co-investment from Western Australian universities. The remaining resources in the Nectar Research Cloud are located in other states and supported by co-investment from universities in those states. Arrangements for access by WA researchers to those resources will clearly require further discussion with all stakeholders of the Nectar research cloud.
It remains the objective of Nectar to provide infrastructure to support the national collaboration needs of Australian researchers and we continue to work with the Pawsey Centre to manage the transition to these new arrangements in Western Australia. Delivering on this collaboration mission continues to rely on the support of our operating partners across Australia and co-investment from Australian research institutions.
The Nectar communication to Research Cloud users provided advice on options for Western Australian researchers to pursue access to cloud resources, including:
- Access to the new Nimbus cloud at the Pawsey Centre;
- Western Australian researchers will continue to access resource allocations granted to collaborating partners in other states.
- Engagement with other node operators to agree access (possibly on a fee-for-services basis)
- Some Research Cloud nodes have offered support for Western Australian participants in research collaborations of interest.
- Western Australian universities may also explore opportunities to continue to invest in access through other Research Cloud node operators.
Nectar continues to provide access to resources for pre-existing allocations by Western Australian researchers at other nodes of the Research Cloud (over 600 virtual CPU cores). Western Australian researchers also continue to have access to the “Project Trial” allocations on the Research Cloud.
The author also notes that: “it would be good if WA could get back its share of the A$61 million to spend on these options”.
Nectar has not withdrawn any of our investment in supporting the needs of Western Australian researchers. Instead, Nectar has invested in cloud computing resources at the Pawsey Centre in Western Australia. Those resources have now been withdrawn from the Nectar Research Cloud and have been redeployed to support the new Nimbus Cloud at the Pawsey Centre. Nectar will continue to work with Western Australian stakeholders to explore the best options for continuing to support Western Australian researcher’s access to the Nectar Research Cloud. Communication with Western Australian research cloud users
The article asserts that “This move has come as a surprise to institutions in WA, who as far as I have been able to tell, did not know that they were about to be cut off” and “The decision raises critical questions about how an entire state of Australia can be cut off from nationally funded research infrastructure?”
No decision has been taken by Nectar to “cutoff an entire state of Australia”. The decision by the Pawsey Centre to withdraw from the Nectar Research Cloud was communicated to Nectar and the users of the Western Australian Nectar node on 2 June 2017. Nectar has continued to work with the Pawsey Centre to manage the transition to the new arrangements in Western Australia; resulting in the communication to Research Cloud users on 4 September.
Commercial Cloud providers and the Nectar Research Cloud: Nectar agrees with the author that commercial cloud providers can and should be leveraged as efficient providers of services to support the needs of Australian researchers. Nectar has engaged with many commercial cloud providers in exploring support for Australian research needs.
We also recognise the value of in-sector cloud resource providers, like the Nectar Research Cloud and the new Nimbus cloud at the Pawsey Centre. In-sector clouds can leverage co-location with other research infrastructure and institutionally held research data. Nectar Research Cloud nodes benefit from co-location with national research networks, national and institutional high performance computing systems and research data repositories.
The value and cost-effectiveness of an in-sector cloud infrastructure has been demonstrated by tens of millions of dollars in additional investment from Australian universities and the emergence of similar science cloud infrastructures around the world.
Arrangements to support the Nectar Research Cloud and recent changes to those arrangements in Western Australia
Since 2012, Nectar has established a national Research Cloud infrastructure to support the collaboration needs of Australian researchers. The Research Cloud has been supported through deployments of Nectar funded computing equipment at eight operating partners across the states of Australia; supported by co-investment from research universities in all the states of Australia.
The objectives of the Research Cloud have been to provide a national interoperable and accessible computational platform on which Australian researchers may collaborate on shared problems across institutional and national boundaries; attracting over 11,000 registered users (over 200 new registered users per month), with thousands of other researchers using services, tools and data deployed on the Research Cloud.
Researchers at all Australian universities have benefited from access to the Nectar Research Cloud since January 2012. The deployment of the Research Cloud infrastructure was completed with the deployment of the Western Australian and New South Wales nodes in December 2014.
In June 2017 the Pawsey Centre advised Nectar of their intention to withdraw the Western Australian node from the Research Cloud. This would include re-purposing of the Nectar funded equipment to support a new national cloud infrastructure service called Nimbus that is integrated with the Pawsey HPC and storage infrastructure. The new cloud component of the national Pawsey service will be independent of the Nectar Research Cloud, but will use the same underlying cloud software (OpenStack), and will align with the recommendations of the infrastructure roadmap for an integrated data-intensive infrastructure system. Nimbus will be available to all Australian researchers using standard merit schemes.
On Monday 4 September Nectar sent a communication (https://nectar.org.au/changes-access-nectar-research-cloud-western-australian-researchers) to all Research Cloud users notifying them of the withdrawal of the Pawsey node of the Research Cloud. The message noted that Nectar is considering the impact on continued access by Western Australian researchers to the cloud resources deployed in other Australian states, given the withdrawal of the capital infrastructure and operating co-investment at the Western Australian node.
The communication includes specific advice on options for Western Australian researchers to continue to receive support, including: the new Nimbus cloud service at the Pawsey Centre and options for access to resources at Nectar Research Cloud nodes in other states. Western Australian research institutions may also explore opportunities to continue to invest in access through other Research Cloud node operators.
It remains the objective of Nectar to provide infrastructure to support the national collaboration needs of Australian researchers and we continue to work with the Pawsey Centre to manage the transition to these new arrangements in Western Australia. Delivering on this collaboration mission continues to rely on the support of our operating partners across Australia and co-investment from Australian research institutions. The Research Cloud has continued to attract additional investment through the node operators and through direct investment in additional capacity by Australian research institutions.
More information about the Nectar Research Cloud is available at https://cloud.nectar.org.au.
For more information more about Nectar see https://nectar.org.au.
The Pawsey Supercomputing Centre declined to provide a statement at this time.