Five years have passed since CUDA, a parallel computing platform and programming environment for GPUs, was released by NVIDIA, the GPU hardware and its software have matured substantially in practical use. Nowadays, GPU computing has been a major topic of research in the HPC area. For linear algebra computations, the GPU acceleration can be widely used not only for compute-intensive operations such as level-3 BLAS, but also for memory-intensive operations such as sparse matrix computation. Although the K computer and the next system do not have accelerators, many core accelerators such as GPUs are still considered as one of the potential technologies for emerging exa-scale supercomputers. In this talk, I will talk about general topics related to the development of linear algebra software on GPUs and some my experiences in this field: (1) overview of recent trends in GPU computing and linear algebra libraries on GPUs, (2) past and future challenges in the development of linear algebra kernels on GPUs, (3) my experience 1: auto-tuning to achieve the best performance, (4) my experience 2: extended-precision arithmetic support on GPUs.