Giving Claude Eyes: Building a World Aware Agent with Model Context Protocol and Local Vision…

Giving Claude Eyes: Building a World Aware Agent with Model Context Protocol and Local Vision Models

In the current landscape of artificial intelligence, we are witnessing a massive shift from simple chatbots to sophisticated agents. However, a fundamental limitation persists: an agent that cannot see and remember its physical environment remains essentially handicapped. As an AI developer focused on creating high-end systems, I recognized that the next frontier is not just about more parameters but about better context. This led me to develop VisionCore MCP, a system that gives Claude a physical presence through a real-time visual feed and a five minute temporal memory using the Model Context Protocol (MCP) and local Vision Language Models (VLMs).

You can access the code from here: Github

The Core Philosophy of Temporal Grounding

Most computer vision systems are inherently stateless. They process a frame, classify an object, and immediately forget it. To build a true agent, I focused on a concept called temporal grounding. This is the ability for an AI to reason about sequences of events over time. If an agent sees a cup on a table at 10:00 AM and notices it is gone at 10:05 AM, it should be able to infer that someone moved it. This requires a chronicle of semantic states rather than a disconnected stream of raw pixels.

The Ingestion Layer: Asynchronous Vision via stream_handler.py

The first major engineering hurdle was the N to 1 threading problem. Standard Python is synchronous, yet video capture must be non-blocking to prevent the entire system from hanging during a slow inference call.

Implementing Background Workers

I developed a StreamHandler class designed to decouple hardware interactions from the core logic.

class StreamHandler:
    def __init__(self, source=0):
        self.cap = None 
        self._lock = threading.Lock() # The Mutex
        
    def start(self):
        # We use CAP_DSHOW for Windows stability
        self.cap = cv2.VideoCapture(self.source, cv2.CAP_DSHOW)
        thread = threading.Thread(target=self._capture_loop, daemon=True)
        thread.start()

In this multithreaded environment, the use of a mutex or threading.Lock() is critical. It prevents race conditions where the camera thread tries to write new data while the AI thread tries to read it simultaneously. This ensures the agent always receives a complete, non corrupted image for processing.

The Reasoning Layer: Orchestrating the Brain with vision_engine.py

For the intelligence of the system, I utilized local models like Llava and Moondream via Ollama. This approach ensures complete privacy since no video data ever leaves the local network. This aligns with my ongoing work in developing video intelligence platforms that prioritize efficient, localized processing.

The Pattern of Semantic State Generation

Instead of overwhelming Claude with raw video data, the system converts pixels into semantic states, which are natural language summaries.

def update_temporal_memory(self, frame_pil):
    # We force the VLM to be literal, not creative
    response = ollama.generate(
        model=self.model_name,
        prompt="Describe the scene in one concise sentence.",
        options={'temperature': 0, 'top_k': 1}
    )
    self.memory.add_state(response['response'])

Small VLMs are often prone to hallucinations. To mitigate this, I set the temperature to zero and top_k to one, forcing the model to be deterministic rather than creative. By using structured analytical prompting, the model is required to describe objective features like shapes and materials before identifying the object itself.

The Memory Layer: Creating the Chronicle with memory_vault.py

The memory vault is where the agent’s temporal context resides. It acts as the short term memory that allows Claude to “remember” what it saw minutes ago.

The Mechanics of Sliding Windows

I employed a collections.deque with time based cleanup logic to manage this memory.

def _cleanup(self):
    now = time.time()
    while self.buffer and (now - self.buffer[0][0]) > self.window_size_seconds:
        self.buffer.popleft()

Using a double ended queue (deque) allows for $O(1)$ complexity in additions and removals. In a real-time system, you cannot afford the performance penalty of rebuilding a list every time a new memory is added. This ensures the agent’s memory remains highly responsive even after hours of operation.

The Interface Layer: Bridging with FastMCP in server.py

The glue for this architecture is the Model Context Protocol. My previous experience presenting on MCP servers has highlighted the importance of clean protocol communication.

Managing JSON-RPC Integrity

A common issue with MCP is JSON-RPC pollution. Since the protocol communicates over standard input and output (stdio), any incidental print statements (like “Success!”) can crash the parser because they do not conform to the expected JSON format.

Implementing IO Redirection

The solution is to redirect all system logs to standard error (stderr).

# Every print in the project looks like this:
print("Initializing...", file=sys.stderr)

Claude ignores stderr during protocol parsing but still captures it in its internal logs. This allows for professional grade debugging and monitoring without compromising the stability of the connection between the AI and the local system.

Real World Optimization and Windows Stability

During the development of VisionCore MCP, I discovered that the default OpenCV backend for Windows, MSMF, was frequently unreliable for background threads. By explicitly switching to CAP_DSHOW (DirectShow), the system achieved significantly faster camera initialization and greater overall stability. This allowed for precise control over resolution, which is vital for balancing AI accuracy with CPU overhead.

Conclusion on the Future of Agentic Vision

VisionCore MCP demonstrates that the next leap in AI utility comes from better context. By providing Claude with a chronicle of the physical world, we move away from static chatbots toward world aware assistants that can truly understand their surroundings.

• The Eyes: Powered by asynchronous OpenCV and protected by Mutex locks.
• The Brain: Driven by local VLMs with deterministic analytical prompting.
• The Memory: Managed by a sliding window deque and persistent SQLite storage.
• The Bridge: Built on FastMCP with redirected logging for protocol safety.

Output:

This architecture sets a new standard for how we integrate physical perception into the reasoning capabilities of large language models.