Most LLM models are at a very high level very similar to Word2Vec in terms of inference on the encoder/non-generative side. Both convert a word/token into a vector representation. GPT does it contextually with all the other words in the input while Word2Vec does it independently for each word. One difference is that a LLM model can also create an "summary" embedding of all the words that's more than just a mean/max of the individual word embedding.

So you can, for example, use a LLM to convert a document into a vector and then also use it to convert a search query into a vector. Then you can find the closest document vectors to the search vector.