Local Agentic LLM Setup

In this note, I'll describe my local agentic LLM setup using Docker Model Runner, Docker Sandbox, and Pi.dev Coding Agent. While the local models may not be as powerful as the frontier models, they are surprisingly good with planning and performing smaller tasks.

Hardware

It is well known that LLMs need a lot of VRAM. I have just 4060 RTX with 8 GB RAM and 32 GB system RAM. A basic gaming setup before LLMs were a thing.

Models

Let's pull the models from Docker Hub. I currently use Qwen 3.6 as it performed better than other models for me. You can use other models as well, the capabilities changes quite fast so there is no universal recommendations.

docker model pull ai/qwen3.6:35B-A3B-UD-Q4_K_M

docker model pull ai/qwen3.5:9B-UD-Q4_K_XL
docker model configure --context-size 8192 ai/qwen3.5:9B-UD-Q4_K_XL

The 9B model fits in to my VRAM so it is the fast default, the 35B with 3B active parameters is much more slower on my system so I use it for more intensive tasks.

Docker Sandbox with Pi Agent

I run agents in sandbox so that I can manage the internet access and have it do permissive tasks. Since my system is limited, I have to use minimal coding agents due to context size limitations.

Dockerfile Dockerfile

# https://hub.docker.com/layers/docker/sandbox-templates/shell
FROM docker/sandbox-templates:shell@sha256:bc721fe1cfddfb234bec0f11c068a38e049cf517e0e278f932b9e241088362f0

RUN npm install -g --ignore-scripts @earendil-works/pi-coding-agent

# Temporarily switch to root to install system dependencies 
USER root

RUN apt update -y && apt install -y --no-install-recommends fd-find ripgrep nano

# Prepare the Agent configurations
RUN mkdir -p /home/agent/.pi
RUN chown -R agent:agent /home/agent/.pi && chmod -R 775 /home/agent/.pi

COPY ./models.json /home/agent/.pi/agent/models.json
COPY ./settings.json /home/agent/.pi/agent/settings.json

# Switch back to agent user
USER agent

models.json json

{
  "providers": {
    "local": {
      "baseUrl": "http://host.docker.internal:12434/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ai/qwen3.6:35B-A3B-UD-Q4_K_M",
          "name": "Qwen 3.6 - 35B"
        },
        {
          "id": "ai/qwen3.5:9B-UD-Q4_K_XL",
          "name": "Qwen 3.5 - 9B"
        }
      ]
    }
  }
}

settings.json json

{
  "defaultProvider": "local",
  "defaultModel": "ai/qwen3.5:9B-UD-Q4_K_XL",
  "defaultProjectTrust": "always"
}

Now we can build and run the sandbox using sbx CLI:

# Build the Template
docker build -t localhost/my-sandbox-template .
docker image save localhost/my-sandbox-template -o my-sandbox-template.tar

# Create the sandbox
sbx template load my-sandbox-template.tar
sbx create --template localhost/my-sandbox-template --name my-sandbox-project shell .
sbx policy allow network localhost:12434

# Start the sandbox
sbx run my-sandbox-project

This setup will mount the local file system, I'll have another article about using the Clone mode so that each sandbox will act as a Git remote locally.