Inference.net

Inference.net provides a distributed inference network for running AI models at scale.

Even though Inference.net is a built-in provider, it still requires you to bring your own key because ngrok.ai inference is not available for it at this time.

Setup

Create an access key

Follow the quickstart to create an access key in app.ngrok.ai.

Store your Inference.net API key

Create a configuration

Create an access key configuration with an inference-net routing rule and allow Inference.net in the access scope.

Assign and send requests

Assign the configuration to your access key and send requests with your access key:

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.ngrok.ai/v1",
    api_key="ng-xxxxx-g1-xxxxx",
)

response = client.chat.completions.create(
    model="inference-net:meta-llama/llama-3.1-8b-instruct/fp-8",
    messages=[{"role": "user", "content": "Hello!"}]
)

Available models

See the model catalog.

Next steps

InceptionLabs Ollama

Concepts

Getting Started With

SDKs

Guides

Observability

Examples

Reference

API Reference

Setup

Available models

Next steps

​Setup

​Available models

​Next steps

Setup

Available models

Next steps