Token Enhancer

A local proxy that strips web pages down to clean text before they enter your AI agent's context window.

One fetch of Yahoo Finance: 704,760 tokens → 2,625 tokens. 99.6% reduction.

No API key. No LLM. No GPU. Just Python.

The Problem

AI agents waste most of their token budget loading raw HTML pages into context. A single Yahoo Finance page is 704K tokens of navigation bars, ads, scripts, and junk. Your agent pays for all of it before any reasoning happens.

The Solution

Token Enhancer sits between your agent and the web. It fetches the page, strips the noise, caches the result, and returns only clean data.

Quick Start

git clone https://github.com/Boof-Pack/token-enhancer.git
cd token-enhancer
chmod +x install.sh
./install.sh
source .venv/bin/activate
python3 test_all.py --live

Usage

As a standalone proxy

source .venv/bin/activate
python3 proxy.py

Then in another terminal:

curl -s http://localhost:8080/fetch \
  -H "content-type: application/json" \
  -d '{"url": "https://finance.yahoo.com/quote/AAPL/"}' \
  | python3 -m json.tool

As an MCP Server (Claude Desktop, Cursor, OpenClaw)

This is the plug and play option. Your AI agent discovers the tools automatically and uses them on its own.

Install the MCP dependency:

source .venv/bin/activate
pip install mcp

Claude Desktop: Add to your config file

Mac: ~/Library/Application Support/Claude/claude_desktop_config.json

Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "token-enhancer": {
      "command": "python3",
      "args": ["/FULL/PATH/TO/token-enhancer/mcp_server.py"]
    }
  }
}

Replace /FULL/PATH/TO/ with the actual path to your clone.

Cursor: Add to .cursor/mcp.json in your project:

{
  "mcpServers": {
    "token-enhancer": {
      "command": "python3",
      "args": ["/FULL/PATH/TO/token-enhancer/mcp_server.py"]
    }
  }
}

Once connected, your agent gets three tools:

fetch_clean fetches any URL and returns clean text (86 to 99% smaller)

fetch_clean_batch fetches multiple URLs at once

refine_prompt optional prompt cleanup, shows both versions so you decide

As a LangChain Tool

from langchain.tools import tool
import requests

@tool
def fetch_clean(url: str) -> str:
    """Fetch a URL and return clean text with HTML noise removed."""
    r = requests.post("http://localhost:8080/fetch", json={"url": url})
    return r.json()["content"]

Add fetch_clean to your agent's tool list. Start python3 proxy.py first.

Features

Data Proxy (Layer 2) Fetches any URL, strips HTML/JSON noise, returns clean text. Caches results so repeat fetches are instant. Handles HTML, JSON, and plain text.

Prompt Refiner (Layer 1, opt in) Strips filler words and hedging while protecting tickers, dates, money values, negations, and conversation references. You see both versions and choose.

MCP Server Plug into Claude Desktop, Cursor, OpenClaw, or any MCP client. Agent discovers the tools and uses them automatically.

API Endpoints (proxy mode)

Run Tests

python3 test_all.py           # Layer 1 only (offline)
python3 test_all.py --live    # Layer 1 + Layer 2 (needs internet)

Roadmap

• Layer 1: Prompt refiner
• Layer 2: Data proxy with caching
• MCP server integration
• LangChain tool example
• Browser fallback (Playwright) for bot blocked sites
• Authenticated session management
• Layer 3: Output/history compression
• CLI tool
• Dashboard UI

Requirements

Python 3.10+. No API keys. No GPU.

License

MIT