Connect to a vLLM Managed Inference endpoint
Send a chat-completion request to a Managed Inference endpoint from the CLI or an OpenAI-compatible client.
Send a chat-completion request to a Managed Inference endpoint from the CLI or any OpenAI-compatible client.
Prerequisites
You need the following before you start:
- A running Managed Inference Job with a serving endpoint. See Create a Managed Inference Job.
- Your Managed Inference API key, if the endpoint requires an authorization header. See Create an API key.
- The CosmicAC CLI installed and configured, for the CLI method. See Install the CLI.
Steps
Find your endpoint name
List your endpoints:
cosmicac models healthcheckEach endpoint appears as Endpoint: <endpoint-name>. Copy the one you want to call.
Send a request
Use the CLI or any OpenAI-compatible client.
If you enabled Require Authorization header when you created the job, you need an API key. If you didn't, answer no to the CLI's API key prompt and omit the Authorization header from the curl request.
Start an interactive chat session:
cosmicac inference chatAnswer these prompts:
- Use an API key: whether to authenticate. Answer yes to be prompted for the key. The CLI masks it as you type.
- Endpoint ID: the endpoint name from the previous step.
The session accepts these commands:
/video <url>: attaches a video to your next message. The URL must point to a video file the model can download in one request. A streaming URL that delivers in chunks, such as a YouTube link, doesn't work./clear-video: removes the attachment./thinkand/no-think: show or hide the model's reasoning in later replies./exitor/quit: ends the session, as does Ctrl+C.