FRONTERA ALLOCATION SUBMISSION GUIDELINES

Introduction

TACC’s Frontera High Performance Computing resource is provided through funding from the National Science Foundation (NSF). Frontera provides a uniquely balanced set of capabilities that supports both capability and capacity simulation, data intensive science, visualization, and data analysis, as well as emerging applications in AI and deep learning.  Representing a substantial public investment, the compute time provided by Frontera is vast, yet still finite, and the process for allocating this time requires rigor to insure the investment is protected and used as effectively as possible. This page defines the policies governing allocations on the Frontera system. Frontera is a system for discovery and new use cases may emerge — please don’t hesitate to contact the project team if you feel you have special circumstances not covered by this document.

Apply for an Allocation

Please thoroughly read this entire page so that you will be prepared to submit a high-quality proposal containing the documentation pertaining to eligibility, a description of the research to be performed using the testbed and sources of reearch support, and a justification for the amount of SUs requested. In addition to the guidelines below, you can view a recording of our webinar "Writing a Successful Frontera Allocation Proposal." You will submit your allocation request via the Resource Allocation System.

  • Only Principal Investigators (PIs) may submit a Frontera allocation request.
  • The quarterly Frontera Pathways and LSCP allocations request periods are: 
    • January 15 - February 15 for April 1 start.
    • April 15 - May 15 for July 1 start.
    • July 15 - August 15 for October 1 start.
    • October 15 - November 15 for January 1 start.
  • The next Frontera LRAC allocation request period will be December 14, 2020 - January 15, 2021 for April 1 start.

Eligibility

Frontera is broadly available to any US researcher (or their non-US collaborators) working in open research. By “open”, the expectation is that any research performed on Frontera will result in publication in a broadly available journal or conference.

An allocation on Frontera is headed by a project PI, typically a faculty member or researcher scientist at a US-based academic institution. The PI may then add additional investigators, including students, to their allocation at their discretion. PIs are responsible for insuring that any users added to their allocation comply with the terms and conditions for use of the Frontera resource. Collaborative projects involving non-U.S. researchers are allowed as long as they include substantive intellectual participation by the U.S. researchers. In joint research projects, foreign collaborators are eligible to make use of that allocation in a manner consistent with the request.

While a PI is typically a research scientist or faculty member at an academic institution, there are a number of other cases where an individual may be eligible to be a PI:

  • NSF Graduate Student Fellows: While in most cases, a graduate student is ineligible to be PI of an allocation request, an exception is made for NSF Graduate Student Fellows. Recipients of these NSF awards can submit requests for Startup allocations. Include supporting documentation (grant number or an award letter) as part of the request submission.

  • Other federal agencies:  Research staff employed by federal agencies or non-NSF FFRDCs are eligible to apply for an allocation if their agency or center does not typically provide research staff with access to resources of adequate scope for the planned research.

  • State educational offices or organizations and local school districts may submit allocation requests intended to broaden the impact, accelerate the pace, and increase the effectiveness of improvements in science, mathematics, and engineering education in both K-12 and post-secondary levels. A teacher or educator at an accredited public or private K-12 school is eligible to apply for an allocation as PI.

  • Independent museums, observatories, libraries, research laboratories, professional societies and similar organizations in the United States that are directly associated with educational or research activities are eligible.

  • U.S. commercial organizations, especially small businesses with strong capabilities in scientific or engineering research or education may apply for an allocation. The Frontera Project is interested in supporting projects that couple industrial research resources and perspectives with those of universities; therefore, it especially welcomes requests from cooperative projects involving both universities and the private commercial sector. It is necessary for these projects to submit their work in an open forum, and make the results of this work readily available to the public.

Allocation Metrics
How is my Time on Frontera Charged?

Compute time on Frontera is allocated in Service Units (SUs). One SU equates to one hour of wall clock time on a single, physical, Frontera node. A typical Frontera node has 56 processor cores and 192GB of RAM. Note that while this is consistent with allocations on most modern systems, it breaks from the historical system of allocating 1 SU per core-hour, so requests should be scaled accordingly. 

Other node types on Frontera are charged as multiples of the basic SU rate.  For instance, nodes with four single-precision GPUs (RTX5000) are charged at 3 SUs per hour (triple the rate of a “normal” compute node). Longhorn nodes with four double-precision GPUs (V100 ) are charged at 6 SUs per hour, reflecting the approximate cost ratio of the GPU nodes to the CPU nodes. Note that this rate is for allocation of the node, not for resources actually consumed – if you run a 2 hour job on V100 GPU node and use only 2 of the 4 GPUs on the node, you are still charged the full rate (12 SUs), as the idle GPUs were not available to another user in this period.

Additional Charges for Advanced Reservations

Advanced Reservations are often necessary to make sure that the system can deliver results in a timely fashion for critical events. However, creating a reservation not only blocks the resource for the time of the reservation (whether it is used or not), it also creates a “shadow” in the schedule, essentially a block of time leading up to the reservation where some compute nodes are kept idle to be sure that they are free at the appointed time.

Reservations can be made with advanced notice.  Reserved nodes are held exclusively for the user/group making the reservation, and will be kept idle if no jobs are submitted.   Reservations can be used through interactive jobs, or through batch jobs (the reservation name must be specified in the batch file); jobs will begin immediately on reserved nodes until all nodes in the reservation are used. Reservations are charged as twice the size and duration of the reservation, minus the cost of jobs actually run through the reservation.   Therefore, a reservation where 100% of the reserved time is used is charged the same rate as if the jobs were submitted to the normal queue.  A reservation that leaves idle resources will be charged at a higher rate.  For instance, a reservation of 100 nodes for one week, where only 60% of the time was used consumed by user jobs, would be charged 2(100 nodes)(168 hours/week) – (0.6100168) = 23,520 SUs.  The same set of jobs run through the normal queue (and subject to wait time) would cost  10,080 SUs.

Allocation Tracks

Frontera allocations are divided into four classes, or “tracks” of allocations through which projects may receive access to portions of Frontera’s available computational resources: Leadership Computing Resource Allocations (which for simplicity we will refer to as “LRACs”), “Pathways” allocations to enable projects to mature into computing at the largest scale, Leadership Community Partnerships (or, LSCPs), and discretionary allocations. The table below provides a preview of the allocation tracks and may (LSCP)

Allocation Track Description
LRAC Large allocations, ready to immediately use up to ~5% of the annual capacity of the system.
Pathways Projects that require, or will soon evolve to require, LRACs but which are not yet ready to compute (effectively) at that scale
Large-Scale Community Partnerships (LSCP) Allocations where we can’t strictly characterize the set of experiments to run in the allocation period, but that line up with a large team or community, a large instrument, or other large NSF investment.
Discretionary Allocations managed directly by the PI for local, educational, industrial (and so on) use that does not fit well into the other three allocation tracks

Leadership Computing Resource Allocations (LRAC)

Description and Need. | Fraction of Total Allocations.| Mechanism of Award. | Allocation size limits:

Ultimately all activities of the Frontera project support the overriding project goal as articulated in the program solicitation: “serve as a national resource for providing predictable and sustained long-term leadership computing capabilities for science and engineering to push the frontiers of knowledge….” In the context of Frontera, we believe this means enabling computation-based scientific discovery, with a particular focus on “big science.”

A minimum of 65% of allocated time on the system will be dedicated to this track.

Potential awardees will submit an allocation proposal to a panel of independent experts that the Frontera team will invite and organize. Recommendations for award will be sent to NSF for their potential concurrence. Deadlines will be at least once per year, possibly twice depending on availability of time. In an attempt to align allocations with research funding, while awards will be made for one year, annual renewals will be available until the end of the associated grant with a simple progress report verifying level of usage, publications, and progress towards research goals.

LRAC requests can be submitted for up to 5M SUs (node hours) per year.  Note that this represents nearly 10% of the full system annual capacity, so we expect typical requests to be much smaller.   The minimum size for a LRAC request is 250,000 SUs per year.

Pathways Allocations

Description and Need. Experience suggests that not all user teams can seamlessly make the transition  from more general-use resources, such as Stampede2, to a system that emphasizes larger scale computing. It is here that latent scaling bugs (such as race conditions, hard-coded array bounds, and so on) are often encountered for the first time. The “Pathways” allocations track is for projects that believe they are ready to begin scaling up, but have not yet fully tested their applications at scale.  Less stringent scaling data will need to be provided to receive a pathways allocation, and the total award size will be smaller, with a focus on preparing to reach the LRAC scale.

Fraction of Total Allocations. Pathways allocations will be available for up to 15% of system capacity.

Mechanism of Award. Pathways award should be submitted through the Frontera portal, and will be awarded quarterly.  The award duration will be for 1 year.

Allocation Size Limits.  Pathways awards may be between 10,000-250,000 SUs per year.

Leadership Community Partnerships (LSCP)

Description and Need. A shortcoming of the current LRAC and XSEDE allocation processes is that the 12-month limitation on allocation requests means that very large science investments by the NSF and other agencies (e.g. MREFC projects like LIGO or LHC) cannot count on being able to “compute to completion.” This mismatch is felt most acutely at the largest computational scale, where researchers are attempting fundamental breakthroughs in their simulations and data analyses. Extended allocations, punctuated by merit reviews to assure appropriate progress, will ensure uninterrupted availability of critical computational resources, and encourage rational allocation of science expertise.

Fraction of Total Allocations. Up to fifteen percent of available system time will be awarded in LSCP allocations, subject to the availability of quality proposals.

Mechanism of Award. Application should be made through the Frontera portal. LSCP awards will be reviewed at least twice per year. The LSCP process will make merit-based allocation awards annually for allocations spanning to three years, with years 2 and 3 awarded after successful completion of a progress review.

Allocation Size Limits.  LSCP awards may be between 25,000-1,000,000 SUs per year.

Discretionary Allocations

The remaining 5% of Frontera’s resources will be allocated by TACC, to be awarded at the PI’s discretion to areas of urgent need, industry projects, or for UT use. To be considered for a discretionary allocation, please contact the project team directly

Education Allocations

Within the discretionary pool, or from unused time in other pools, time on Frontera may be awarded for classroom, training, or other educational uses.  TACC reserves the right to direct these to other computing resources if it deems appropriate.  An Educational allocation may request up to 15,000 SUs. This would be approximately equivalent to a 50 student class allocating 8 nodes per student for a 40 hour work week. To request an Education Allocation please contact the project team directly.

Startup Allocations

A startup allocation may be requested to gather benchmark data for a larger allocation request, or to test the suitability of the system for a particular research project.  Startup requests may be for up to 5,000 SUs for Frontera CPU nodes, 3,000 SUs for Frontera GPU nodes, and 6,000 SUs for Longhorn GPU nodes. To request a Startup Allocation please contact the project team directly.

In the event that the quality of applications in the allocation tracks does not result in award of the full allocation, the unused portion will be offered through XSEDE as an L2 resource. As a partner in XSEDE, we have all the mechanisms in place to do this.

Proposals and Review Criteria

Frontera allocation are reviewed for merit by a committee consisting of members of the resource allocation board. The board is made up of researchers in the academic community with support from members of the project team. The board makes recommendations to the project based solely on the merit of the proposal, and not on overall availability of the resource. The board will review the proposal and make a recommendation to the project team based on the following criteria:

  • Intellectual Merit: Is the proposed work supported by a grant or grants that have undergone review for intellectual merit and/or broader impact (if relevant to the allocation)? If so, is the allocation request consistent with the objectives of the supporting grants and is the scale of resource use commensurate with the level and purpose of support? If a request is not supported by a merit-reviewed award, reviewers will assess the intellectual merit of the proposed work and factor that into their overall recommendation. Reviewers will also consider whether the identified support provides necessary and sufficient human resources to complete the proposed work.

  • Appropriateness of Methodology: Does the request describe appropriate tools, methods, and approaches for addressing the research objectives? These methodologies may be community codes or models, data analysis methods, or algorithmic formulations expressed in user-developed scripts or tools. Is this methodology appropriate for use on Frontera?

  • Appropriateness of Research Plan: Does the request describe necessary and sufficient computational experiments to answer the research questions posed? In some cases, the research plan may be more reasonably expressed as estimates of resource use, supported by past or early experience. Serious concerns about the research plan will be documented in reviews and may lead to reduced allocation awards.

  • Efficient Use of Resources: Has the request identified appropriate resources for undertaking the research plan with the proposed methodology? And will those resources be used as efficiently as is reasonably possible?

  • For renewal requests, success of prior or other existing allocations in terms of significant results, published research and new funding.

  • Demonstrated need for a large scale resource.

Proposals that are deemed competitive by the project advisory board are then reviewed by the project team for technical feasibility (i.e, can the project be implemented in the Frontera environment?) and for the availability of time on the resource.  Recommended proposals will then be sent to NSF for final approval.  

For LRAC Allocations, proposals will be reviewed at least twice per year. Startup and Educational Allocation requests receive a fast track review, bypassing the project advisory board.

Conflict of Interest and Confidentiality

Every effort is made to avoid conflicts of interest. Reviewers are not allowed to review or be present for the discussion of requests from their home institution, former students, postdocs, advisors, or current and recent collaborators. If in the opinion of a PI, a certain individual has a conflict of interest, the PI may request that the individual not act as reviewer on their request.   All reviews remain confidential, and are made available only to:  The PI and Co-PIs submitting the request, assigned reviewers from the Advisory Committee, project staff involved in the allocation review process, and NSF program officers for the project.

While the contents of reviews and details of allocation requests and experiments remain confidential, a list of projects receiving allocations and the general area of research may be made public on the project web site and used in project reports and presentations.

Request for an Allocation Advance.

In some cases, a researcher might exhaust the available allocation in their Startup Allocation, and have a pressing need for additional allocation before their Research Allocation proposal can be reviewed. Similarly, researchers might have pressing needs while a renewal request is pending. In these instances, users can request an advance against a pending allocation proposal for up to 90 days. Advance request can be for up to 20% of the SUs requested in the proposal.

Supplemental Requests

A supplement is a request for additional resources during an existing allocation’s one year time frame. Its purpose is to support changes in the original computational research plan that are required to achieve the scientific goals of the project. This may include altered or new goals, or support for projects proceeding more rapidly than anticipated or that require more resources than anticipated. Supplement awards are highly dependent upon availability of resources and limited to when allocation awards at the previous allocation period have been reduced to eliminate oversubscriptions. Supplements are not a mechanism to acquire additional resources for awards that were recommended for less than the amount originally requested.   Supplemental requests require the filing of a progress report that documents what has been done with the originally awarded allocation, and makes the case for additional resources.

Extension Requests

Extensions of allocation periods beyond the normal 12-month duration can be requested through the portal. This request brings no new allocation, but keeps unused allocations from expiring. A brief reason is required for not using the awarded allocation, but no formal documentation is needed. Note that granting extension requests reduces the amount of allocation that can be given to users in the next year. Extension requests are subject to NSF concurrence and are not guaranteed. Also, please be aware that you cannot submit an extension request on a project for which a renewal has been submitted (or for which you intend to submit a renewal request).

Proposal Format

LRAC, Pathways, and LSCP Submissions Requests

When writing your LRAC request, be clear and concise. We strive to have domain experts review every request, but they may not have deep expertise in your specific subdomain. Someone outside of your area should be able to understand the scientific objectives and understand why the chosen technique is preferred over another.

Renewal requests require less documentation, as described later (jump link to section).

The documents required of a new LRAC request ensures reviewers can effectively determine how each request satisfies the Frontera allocation Review Criteria. LRAC requests are limited to 10 pages.

Research requests must include a well-documented resource-use plan that describes how the requested allocations are necessary and sufficient to accomplish the project’s research objectives. An effective resource-use plan must address the Frontera allocation Review Criteria and, in particular, must include the following elements:

  • Scientific Background
  • Research Objectives and specific questions to be pursued
  • Resource Usage Plan to achieve the research objectives
  • Justification of the allocation amounts for all resources and resource types
  • Access to other CI resources and why those resources are not available or sufficient for the work proposed in this request

Scientific Background and Support

Succinctly state the scientific objectives that will be facilitated by the allocation. The existing merit-reviewed supporting grants should be listed and briefly described; please include the end date for each grant listed. Requests with merit-reviewed supporting grants will not be subject to further scientific review by the LRAC; however, the stated scientific objectives must match or be sub-goals of those described in the listed funding award(s). The description of the objectives should be sufficient to allow the reviewers to assess the resource usage plan.

Research Questions

Identify the specific research questions that are covered by the allocation request. Within the context of the scientific background and supporting grants, these objectives and questions should be stated so that the reviewers can understand how elements of the resource plan will contribute to relevant answers. If you consider the allocation request in terms of computational experiments, the research objectives and questions define the experiments to be conducted.

Resource Usage Plan

The bulk of the document should focus on the resource usage plan and allocation request. Inadequate justification for requested resources is the primary reason for most reduced or denied allocations. Once again, the PI should keep in mind the Frontera allocation Review Criteria when describing and justifying the choice of resources and allocation amounts.

Justifying Allocation Amounts

This section should contain code performance timings, resource usage details, and scaling information to support the calculation of the resource request. Ideally, the code performance data should be provided from benchmark runs on the resource requested, using the model configuration(s) needed for the computational plan proposed and demonstrating the scaling efficiency to the job size(s) planned. If applicable, the panel may accept well-justified performance data from architecturally equivalent systems. PI’s should contact the Frontera Allocations Coordinator to obtain a small allocation for benchmarking or to discuss non-Frontera performance data to be used in their submission.

The justification of allocation should take the quantitative parameters from the resource usage plan and combine them with benchmark cost information to calculate the allocation needed. Where possible for computational experiments, the justification should tabulate and calculate the costs of conducting each experiment for each resource and resource type.

Disclosure of Access to Other Compute Resources

Because of the high demand for NSF-funded resources, Frontera and the LRAC closely consider whether the PI has access to other, less constrained resources on which the proposed work could be conducted. All PIs must describe their local resources and other non-Frontera resources available to them, including both local and other national resources.

Page limit: 10 pages

References

A PI may use this OPTIONAL document to separate a lengthy bibliography of cited work to take full advantage of the page limits in the Main Document. This bibliography is not the publications resulting from prior Frontera support, which should be entered into the Frontera publications database, but rather other citations referenced in describing or supporting the intellectual merit of the proposed work or the appropriateness of the proposed approach for addressing the research objectives.

Page limit: No limit.

Renewal Submission Requirements

For science teams that are renewing their Frontera LRAC allocation to continue the same research topic, a progress report may be submitted in lieu of a new request. The progress report should describe:

  • How the PI’s current or prior allocation was used and summarize the findings or results.
  • A list of any relevant  publications and other communications resulting from the work.
  • Explanation of  any significant digressions from the originally proposed resource usage plan for the prior period.  Include an updated resource usage plan for the upcoming year, and a justification of the allocation amounts. Previous benchmarking data from the original request can be referenced.
  • If a PI’s allocation was significantly underutilized, the Progress Report should briefly describe the reasons for the underutilization and any mitigation by the PI to avoid the situation with the current allocation.

Page limit: 8 pages.

A Startup Allocation request consists of the following components:

  • PI Name, Title, and Organization
  • (optional) Co-investigator names, titles, and organizations
  • Summary of the proposed experiments (1 page maximum).
  • Estimated resources required (include a short justification of the number and type of nodes you will request, as well as any storage requirements). (1 page maximum)
  • List any sources of funding (or pending funding) that support the proposed research, including funding agency and grant name(s) (Startup Allocations may be for unfunded exploratory research)

An Education Allocation request consists of the following components:

  • PI Name, Title, and Organization
  • (optional) Co-investigator names, titles, and organizations
  • Summary of the proposed course or training session to be supported, including the expected number of students who will participate (1 page maximum).
  • Estimated resources required (include a short justification of the number and type of nodes you will request, and any specific software, hardware, or networking requirements. (1 page maximum)
  • If applicable, list any sources of funding that support the proposed course, including funding agency and grant name(s) (Educational Allocations do not require any external funding, but in cases where it does exist, the project would like to record any grants that are impacted by Frontera).

Document Formatting

While readability is of greatest importance, documents must satisfy the following minimum requirements. Documents that conform to NSF proposal format guidelines will satisfy these guidelines.

  • Margins: Documents must have 2.5-cm (1-inch) margins at the top, bottom, and sides.
  • Fonts and Spacing: The type size used throughout the documents must conform to the following three requirements:

Use one of the following typefaces identified below:

  • Arial 11, Courier New, or Palatino Linotype at a font size of 10 points or larger;
  • Times New Roman at a font size of 11 points or larger; or
  • Computer Modern family of fonts at a font size of 11 points or larger.

A font size of less than 10 points may be used for mathematical formulas or equations, figures, table or diagram captions and when using a Symbol font to insert Greek letters or special characters. PIs are cautioned, however, that the text must still be readable.

Page Numbering: Page numbers should be included by the submitter.

One Project per PI

An individual may be a PI on only one active Frontera allocation request/project at a given time. Several distinct research activities can be combined in a single resource request, however, the resource request for each activity must be justified, and any allocation-size limits apply to the aggregate request.

The single-project rule is designed to minimize the effort required by PIs for submitting resource requests and the overhead to the process for reviewing those requests. While PIs may have several different funded grants that require computational support, these should be included as sub-projects within a single request.

There are several exceptions to this rule:

  • A PI may submit a Research request at any time during an active Startup allocation. A successful Research allocation will supersede any Startup allocations and start a new 12-month allocation period.

  • If a PI is leading an individual research activity as well as a community consortium activity or community service activity, separate requests for these distinct activities will be considered.

  • A PI with active Startup or Research projects may submit requests for any number of Education projects to keep classroom and training users separate from research activities.

Similarly, to minimize the effort required to gain access to Frontera, closely collaborating researchers should submit a single collaborative request rather than several individual requests. For example, a PI and associated post-doctoral researchers; investigators supported by the same funding grant; and researchers in the same lab group should consider submitting a request describing and justifying the various subactivities. One of the collaborators is designated as the PI, and others can be designated as co-PIs.

Acknowledgement of Support

An acknowledgement of support from the Frontera project and the National Science Foundation should appear in any publication of material, whether copyrighted or not, that describes work which benefited from access to TACC’s Frontera resource. For suggested language, see TACC Citation

Other Requests and Feedback

The project team is interested in receiving as many excellent requests as possible, and encourages any potential investigators who wish to use the resource to not hesitate in contacting the team for additional help in preparing this request, with questions about the process, or to discuss any type of project that may not fit the structure described here.   The project team also welcomes input in improving this process, and encourages feedback on both this document and the process.