Deploying an LLM Inference Server
For better or worse, the age of the Large Language Model (LLM) has officially arrived. With it comes fresh new opportunities and problems for humanity, such as increased worker productivity alongside massively increased energy demand and job displacement. The widespread adoption of LLMs from just a few key suppliers brings other issues, such as vendor dependence and data security concerns. Besides this, private research and development of these models continues at a blistering pace. Because of the needs of researchers and private vendors, and perhaps also the other issues, a new trend has emerged whereby private instances of LLMs are deployed locally on personally owned or rented infrastructure. New software stacks, such as the open source vLLM, or Nvidia's proprietary NIM, have appeared to fill the niche of LLM deployment and administration.
In this article, we explore the process of deploying a private LLM inference server using Nvidia Run:ai on Imperial College London's new HX3 cluster for AI computing. There are several reasons we might want such a thing. Perhaps we wish to automate some menial tasks that are too difficult to write scripts for, or use it as a personal code reviewer when using sensitive data or software. Maybe we want it to help us stay organised in our workplace, or monitor some critical system. Maybe we just want it to chat with. All of these are made possible with a private inference server once you understand how to use its Application Programming Interface (API), and understand how external tools like file readers are exposed to the LLM. All of this and more will be discussed in this guide, along with step-by-step instructions to follow along with, so keep reading if that sounds like your thing.
