❯ Don't Share Your Password with AI: Build Your Own Agent System

by Jonas Reyes — the builder's desk, Side Quest Studios
October 31, 2026

Alright, listen up. It's October 2026. If you're still reading vendor whitepapers about "synergistic AI transformations" or thinking your company's "AI strategy" involves forwarding emails to ChatGPT, you're already behind. Two years from now, you'll either be running real systems or you'll be asking why your "AI assistant" just drained your bank account. Bank of America had to tell people, kindly, not to share their banking passwords with AI Bank of America. Seriously? We're still at this level?

This isn't about being paranoid. It's about owning your stack. It's about control. It’s about not letting a black box with a slick UI become the weakest link in your operational security. You don’t give your root password to a SaaS vendor, do you? No. So why would you give your critical credentials to an LLM you don't control, let alone one that might be running on shared infrastructure?

We're going to talk about building your own agent system, one that's secure, auditable, and actually works without needing a corporate warning to prevent self-immolation.

The Problem with Black Box Agents: Trust and Data Exfiltration

The core issue is trust, or rather, the lack thereof. When you use a third-party AI agent, you're not just trusting their model; you're trusting their entire pipeline, their data handling, their security practices, and every single dependency in their stack. Anthropic's IPO filing alone listed weapons, blackmail, and rogue agents as risks Anthropic IPO. That's not FUD; that's their own risk assessment. If they're worried, you should be too.

Your data, your credentials, your business logic — it all flows into their system. What happens if their API is compromised? What if a rogue engineer (or even a well-intentioned but careless one) pushes a bad update? What if your "private" data is subtly used to train their next model? You have no visibility. Zero.

The Solution: Local Control, Open Source, and Python

The answer is simple: build it yourself. Not from scratch, obviously. Leverage the excellent open-source tools already out there. This isn't about reinventing the wheel; it's about owning the axle.

We're talking Linux, Python, and a clear, modular architecture. This gives you:

  1. Visibility: You see every line of code, every API call, every data flow.
  2. Control: You decide what data goes where, what models are used, and what permissions are granted.
  3. Security: You manage the environment. You patch it. You harden it.
  4. Adaptability: As new models and techniques emerge, you can swap them in without migrating your entire operation to a new vendor.

Step 1: Isolate Your Agent Environment

First, create a dedicated user and a virtual environment. Don't run your agent as root, ever. Don't pollute your global Python installation.

sudo useradd -m -s /bin/bash ai_agent
sudo passwd ai_agent # Set a strong password
sudo usermod -aG sudo ai_agent # If you absolutely need sudo for specific tasks, otherwise omit

# Switch to the new user
su - ai_agent

# Create and activate a virtual environment
python3 -m venv ~/agent_env
source ~/agent_env/bin/activate

Now, all your agent's dependencies and code live within ~/agent_env, isolated from the rest of your system.

Step 2: Choose Your LLM Backend Wisely

You need an LLM. This is where most people get tripped up. Forget the shiny web UIs for a moment. You need an API.

OpenAI/Anthropic/Google API: If you must use a commercial model, use their API directly. Configure strong API key management (rotate keys, restrict IP access, use environment variables, never hardcode). ```python import os import openaiopenai.api_key = os.getenv("OPENAI_API_KEY")

... your agent logic using openai.Completion.create or chat.Completion.create

* **Local LLM (Preferred):** For true control and data privacy, run your own. This is where it gets interesting. * **Ollama:** For quick local deployment of many models.bash curl -fsSL https://ollama.com/install.sh | sh ollama run llama3 # Downloads and runs Llama 3 Then, your Python agent can interact with it via its local API (default `http://localhost:11434/api/generate`). * **vLLM/Text Generation Inference (TGI):** For serious production-grade serving on GPUs. These require more setup but offer massive throughput. You'd typically run these in Docker containers.bash # Example for vLLM with Docker (assuming NVIDIA drivers and toolkit installed) docker run --gpus all --shm-size 1g -p 8000:8000 vllm/vllm:latest --model meta-llama/Llama-2-7b-hf `` Your agent then hitshttp://localhost:8000/v1/completions`.

The point is: you control the binary, you control the hardware, you control the data flow.

Step 3: Architect Your Agent with Tools, Not Passwords

Your agent should never hold your banking password. Period. Instead, it should orchestrate tools. Think of your agent as a fancy bash script that can reason.

A "tool" is a function or script that performs a specific action. The agent’s job is to decide when to call which tool and with what arguments.

Example: Instead of "AI, pay my bills," it's "AI, use the pay_bill tool with amount=200, recipient='Electric Co', account_id='12345'."

# tools.py
import subprocess

def get_current_stock_price(symbol: str) -> str:
    """Fetches the current stock price for a given symbol using a CLI tool."""
    try:
        # Example: using a hypothetical 'stock_fetcher' CLI tool
        result = subprocess.run(['stock_fetcher', symbol], capture_output=True, text=True, check=True)
        return result.stdout.strip()
    except subprocess.CalledProcessError as e:
        return f"Error fetching stock price: {e.stderr}"

def write_to_database(table: str, data: dict) -> str:
    """Writes a dictionary of data to a specified database table."""
    # This function would use a secure database connection (e.g., psycopg2 for PostgreSQL)
    # and properly parameterized queries, never f-stringing user input into SQL.
    # For illustration, let's pretend it's a secure ORM call.
    print(f"Attempting to write to {table}: {data}")
    # In reality: db_session.add(YourTable(data)); db_session.commit()
    return f"Data successfully written to {table}."

# Define your tools for the agent
available_tools = {
    "get_current_stock_price": get_current_stock_price,
    "write_to_database": write_to_database,
    # Add more tools: send_email, update_calendar, run_system_command_safely, etc.
}

Your agent's main.py would then expose these tools to the LLM (e.g., via OpenAI's function calling API or a custom prompt for local models) and execute them based on the LLM's output.

Crucially, each tool should have minimal, specific permissions. pay_bill should only be able to initiate a payment, not browse your entire financial history. This is classic Unix least privilege principle.

Step 4: Secure Your Credentials (No Plaintext!)

Tools often need credentials. Don't hardcode them. Don't put them in .env files that get committed to Git.

  • Environment Variables: Best for ephemeral secrets, passed at runtime. bash export DB_PASSWORD="very_secure_password" python3 main.py
  • HashiCorp Vault / AWS Secrets Manager / GCP Secret Manager: For production. Your agent authenticates to the secret store, fetches the secret, and uses it. This is the only way to manage secrets at scale without losing your mind or getting breached. python # Example using a hypothetical 'get_secret' function # In reality, this would involve client libraries for Vault, etc. db_password = get_secret("my_app/db_creds", "password")
  • Keyring (Linux): For desktop-like environments where an agent might need to access user-specific credentials. python-keyring library is your friend.

Step 5: Implement Logging and Auditing

If you can't log it, you can't debug it. If you can't audit it, you can't secure it. Every action your agent takes, every tool it calls, every LLM prompt and response – log it.

import logging
import json

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def log_agent_action(action_type: str, details: dict):
    logging.info(f"AGENT_ACTION - Type: {action_type}, Details: {json.dumps(details)}")

# In your agent's loop:
# ... after LLM decision
log_agent_action("LLM_DECISION", {"tool_name": tool_name, "tool_args": args})
# ... before tool execution
log_agent_action("TOOL_EXECUTION_START", {"tool_name": tool_name, "args": args})
try:
    tool_output = available_tools[tool_name](**args)
    log_agent_action("TOOL_EXECUTION_SUCCESS", {"tool_name": tool_name, "output": tool_output})
except Exception as e:
    log_agent_action("TOOL_EXECUTION_FAILURE", {"tool_name": tool_name, "error": str(e)})

Ship these logs to a central logging system (e.g., rsyslog to a central ELK stack or Loki). This gives you a complete, immutable trail of what your agent did, when, and why. Essential for debugging, security, and compliance.

Step 6: Deploy with Systemd

Once your agent is tested and running, manage it like any other critical service. systemd is your workhorse.

Create a service file: /etc/systemd/system/ai_agent.service

References

💬 Discuss this article in the community — 0 replies →
Comments live on community.sqs.chat — one thread per article.