The exponential growth of floating point power in GPUs, combined with high memory bandwidth, has given rise to an attractive platform upon which to deploy HPC applications. We review the QUDA library which is a domain-specific library designed to accelerate legacy lattice quantum chromodynamics application through providing a rich library of the common performance-critical algorithms, including highly optimized sparse linear solvers. QUDA is a toolbox in which to research and develop algorithms for the exascale future where flops are free and memory bandwidth determines performance. We describe the techniques currently present in QUDA and also consider future directions for development.