CPU Security Camera with AI Vision-Language Analysis

A local, real-time security camera system that runs entirely on CPU — no GPU required. It combines YOLOv8n object detection with a tiny Vision-Language Model (SmolVLM-256M or Florence-2-base) to detect, crop, analyze, and alert on targets via Telegram.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                        main.py (Event Loop)                         │
│                                                                     │
│   ┌──────────┐    ┌──────────┐    ┌────────────────────────────┐   │
│   │ RTSP     │───▶│ YOLOv8n  │───▶│ Target detected?           │   │
│   │ Camera   │    │ (CPU)    │    │ conf > 0.70 + cooldown OK  │   │
│   └──────────┘    └──────────┘    └────────────┬───────────────┘   │
│                                                 │                   │
│                                        ┌────────▼────────┐         │
│                                        │ Crop-First ROI  │         │
│                                        │ 20% padding     │         │
│                                        │ clamp to bounds  │         │
│                                        └────────┬────────┘         │
│                                                 │                   │
│                                        ┌────────▼────────┐         │
│                                        │ Background      │         │
│                                        │ Thread (daemon)  │         │
│                                        └───┬─────────┬───┘         │
│                                            │         │              │
│                                   ┌────────▼──┐ ┌───▼──────────┐  │
│                                   │ Tiny VLM  │ │ cv2.imshow() │  │
│                                   │ Inference │ │ Live Feed    │  │
│                                   └────────┬──┘ └──────────────┘  │
│                                            │                       │
│                                   ┌────────▼──────────┐           │
│                                   │ Telegram Alert    │           │
│                                   │ Photo + Caption   │           │
│                                   └───────────────────┘           │
└─────────────────────────────────────────────────────────────────────┘

Key principle: The main video loop never blocks. YOLO detection runs every frame; when a target is found, a daemon thread handles VLM inference and Telegram alerting independently.

---

Features

• CPU-Only Execution — All PyTorch models forced to device='cpu' with torch.float32
• Crop-First Technique — Only the detected region (with 20% padding) is resized to 512x512 for VLM analysis — never the full 1080p frame
• Dual VLM Support — Choose between SmolVLM-256M-Instruct (256M params, 512x512) or Florence-2-base (230M params, 336x336)
• Non-Blocking Pipeline — Background threading.Thread(daemon=True) ensures the live feed never freezes
• Configurable Cooldown — Prevents alert floods (default 10 seconds)
• Telegram Alerts — Sends cropped snapshot with AI-generated description as a photo message
• Robust Error Handling — Background threads catch all exceptions; the main loop never crashes

---

File Structure

.
├── .env                  # Environment configuration (RTSP, Telegram, model)
├── requirements.txt      # Python dependencies
├── config.py             # Loads and validates .env variables
├── alert_manager.py      # TelegramAlerter class — sends photos with captions
├── ai_brain.py           # TinyCPUAnalyzer class — VLM inference on CPU
└── main.py               # Main entry point — hybrid async pipeline

---

Installation

Prerequisites

• Python 3.11 or higher
• A working camera (USB, IP/RTSP, or webcam)
• Internet connection (for initial model download from HuggingFace)
• A Telegram bot (for alerts)

Step 1: Clone or Download

git clone <your-repo-url>
cd cpu-security-camera

Step 2: Install Dependencies

pip install -r requirements.txt

This installs:

Package	Purpose
opencv-python	Video capture and display
ultralytics	YOLOv8n object detection
torch	PyTorch CPU inference
transformers	HuggingFace model loading (SmolVLM / Florence-2)
Pillow	Image format conversion
requests	Telegram Bot API calls
python-dotenv	.env file loading
accelerate	Model loading optimizations

Step 3: Configure Environment

Copy and edit the .env file:

# .env
RTSP_URL=rtsp://admin:password@192.168.1.100:554/stream1
TELEGRAM_BOT_TOKEN=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
TELEGRAM_CHAT_ID=987654321
COOLDOWN_SECONDS=10
TARGET_CLASSES=person,dog,car
AI_MODEL=smolvlm

---

Configuration Reference

Environment Variables

Variable	Required	Default	Description
RTSP_URL	Yes	—	RTSP stream URL for your camera. Also supports USB webcams via 0.
TELEGRAM_BOT_TOKEN	Yes	—	Bot token from @BotFather.
TELEGRAM_CHAT_ID	Yes	—	Your Telegram chat ID (use @userinfobot to find it).
COOLDOWN_SECONDS	No	10	Minimum seconds between alerts to prevent flooding.
TARGET_CLASSES	No	person,dog,car	Comma-separated COCO classes to detect.
AI_MODEL	No	smolvlm	VLM to use: smolvlm or florence2.

Supported Target Classes

Only these COCO classes are mapped (YOLOv8n is trained on COCO 80-class):

Class	COCO ID
person	0
car	2
dog	16

You can extend the COCO_LABELS dictionary in main.py to add more classes (e.g., bird = 14, cat = 15, etc.).

VLM Model Comparison

	SmolVLM-256M-Instruct	Florence-2-base
Parameters	256M	230M
Input Size	512×512	336×336
Prompt Style	Free-text question	<MORE_DETAILED_CAPTION>
Best For	Descriptive analysis	Detailed captions
Load Time	~15-30s (first run)	~15-30s (first run)
Inference Speed	~3-8s per crop on CPU	~4-10s per crop on CPU

First run downloads the model from HuggingFace (~500MB-1GB). Subsequent runs load from cache.

---

Usage

Quick Start

python main.py

What Happens

1. The system connects to your RTSP camera
2. YOLOv8n loads on CPU and begins detecting targets every frame
3. A live feed window named "Security CPU Feed" opens
4. When a person, dog, or car is detected (confidence > 0.70):
   • A green bounding box is drawn on the live feed
   • The ROI is cropped with 20% padding
   • A background thread runs VLM analysis on the crop
   • A Telegram alert is sent with the photo and AI description
5. The cooldown timer prevents repeated alerts for the same target

Controls

Key	Action
q	Quit the application
Ctrl+C	Graceful shutdown

Console Output Example

[14:30:01] INFO Config loaded:
[14:30:01] INFO RTSP_URL=rtsp://admin:***@192.168.1.100:554/stream1, COOLDOWN=10s, TARGETS=['person', 'dog', 'car'], AI_MODEL=smolvlm
[14:30:01] INFO [INIT] Monitoring COCO class IDs: {0: 'person', 2: 'car', 16: 'dog'}
[14:30:01] INFO [CPU] Loading YOLOv8n on CPU...
[14:30:02] INFO [CPU] YOLOv8n ready.
[14:30:02] INFO [INIT] Loading AI brain (smolvlm)...
[14:30:05] INFO [AI] Loading model 'HuggingFaceTB/SmolVLM-256M-Instruct' on CPU (float32)...
[14:30:20] INFO [AI] Model loaded successfully on CPU.
[14:30:20] INFO [TELEGRAM] Bot connected: @my_security_bot
[14:30:20] INFO [INIT] Connecting to camera: rtsp://...
[14:30:20] INFO [INIT] Camera connected.
[14:30:25] INFO [CPU] YOLO detected person (conf=0.85). Cropping and queuing AI...
[14:30:31] INFO [AI] Analysis (5.8s): "Person standing near door holding a flashlight."
[14:30:31] INFO [TELEGRAM] Alert sent successfully.
[14:30:31] INFO [CPU] Alert sent.

---

Telegram Setup

Create a Bot

1. Open Telegram and search for @BotFather
2. Send /newbot
3. Choose a name (e.g., "Security Camera Alert Bot")
4. Choose a username (e.g., my_security_cam_bot)
5. Copy the bot token (looks like 123456789:ABCdefGHIjklMNOpqrsTUVwxyz)

Get Your Chat ID

Option A — Personal chat:

1. Search for @userinfobot on Telegram
2. Send any message
3. It replies with your Chat ID

Option B — Group chat:

1. Add your bot to a group
2. Send a message in the group
3. Visit: https://api.telegram.org/bot<YOUR_TOKEN>/getUpdates
4. Find the chat.id field in the response (it will be negative for groups)

Test the Bot

curl "https://api.telegram.org/bot<YOUR_TOKEN>/sendMessage?chat_id=<CHAT_ID>&text=Test+message"

---

How the Crop-First Technique Works

The core optimization that makes CPU analysis feasible:

Original Frame: 1920×1080 (2,073,600 pixels)
                    │
                    ▼
        ┌─────────────────────┐
        │  YOLO Detects:      │
        │  person at          │
        │  (x1=400, y1=200,  │
        │   x2=600, y2=700)  │
        └─────────┬───────────┘
                  │
                  ▼
        ┌─────────────────────┐
        │  Add 20% Padding:   │
        │  cw = 200, ch = 500 │
        │  px = 40, py = 100  │
        │  Crop: 280×700 px   │
        └─────────┬───────────┘
                  │
                  ▼
        ┌─────────────────────┐
        │  Resize Crop to:    │
        │  512×512 (SmolVLM)  │
        │  336×336 (Florence) │
        │  = 262,144 pixels   │
        └─────────┬───────────┘
                  │
                  ▼
        ┌─────────────────────┐
        │  VLM Inference on:  │
        │  512×512 crop only  │
        │  (NOT full frame)   │
        └─────────────────────┘

Full frame: 2,073,600 pixels → Crop: 262,144 pixels = 87% fewer pixels

Result: The VLM processes 87% fewer pixels than a full-frame resize, while focusing on the highest-detail region of the target.

---

Thread Safety and Error Handling

Main Loop Guarantees

• The while True loop never waits for VLM inference
• Each alert spawns a new daemon thread — threads are fire-and-forget
• cv2.waitKey(1) always executes at display refresh rate
• Camera reconnection: if cap.read() fails, sleeps 2s and retries

Background Thread Protection

def run_ai_analysis(crop, class_name, ai, alerter):
    try:
        description = ai.analyze_crop(crop, detected_class=class_name)
        alerter.send_alert(crop, description)
    except Exception as exc:
        logger.error("[AI] Background thread error: %s", exc, exc_info=True)

• All exceptions are caught and logged — the main loop is unaffected
• Empty crops are rejected before inference
• Memory errors, model failures, and network issues are handled gracefully

Resource Cleanup

try:
    while True:
        ...
finally:
    cap.release()
    cv2.destroyAllWindows()

Guaranteed cleanup on Ctrl+C, q press, or any unhandled exception.

---

Performance Tuning

Reduce CPU Load

Tweak	Effect
Lower YOLO confidence threshold	Fewer false positives, but may miss targets
Increase COOLDOWN_SECONDS	Fewer AI inferences per minute
Use AI_MODEL=florence2	Slightly smaller model (230M vs 256M)
Reduce VLM max_new_tokens	Faster inference, shorter descriptions

Increase Detection Accuracy

Tweak	Effect
Use yolov8s.pt or yolov8m.pt	More accurate detection (slower on CPU)
Lower confidence to 0.50	Catches more targets
Add more classes to COCO_LABELS	Expands what the system monitors

Frame Rate Optimization

The YOLO inference runs on every frame. To reduce CPU usage:

# In main.py, skip frames:
if frame_count % 2 == 0:  # Run YOLO every other frame
    continue

---

Troubleshooting

Problem	Solution
Cannot open video stream	Check RTSP_URL. For USB webcams, use 0 instead of an RTSP URL.
TELEGRAM_BOT_TOKEN invalid	Verify token with curl https://api.telegram.org/bot<TOKEN>/getMe
TELEGRAM_CHAT_ID not found	Message @userinfobot or check getUpdates API
Model download is slow	First run downloads from HuggingFace. Use a fast connection.
torch.cuda.OutOfMemoryError	Should not happen on CPU. If it does, reduce max_new_tokens.
Analysis takes too long	SmolVLM: ~3-8s. Florence-2: ~4-10s. This is normal for CPU-only.
Live feed is choppy	YOLOv8n on CPU targets ~5-15 FPS. Reduce resolution or use frame skipping.
ImportError: No module named 'ultralytics'	Run pip install -r requirements.txt

---

CPU Performance Benchmarks

Typical performance on common hardware:

Hardware	YOLO FPS	VLM Inference (SmolVLM)	VLM Inference (Florence-2)
Intel i7-12700	12-18	4-6s	5-8s
AMD Ryzen 7 5800X	10-16	5-7s	6-9s
Intel i5-10400	6-10	6-9s	7-11s
Apple M1 (via MPS)	15-22	3-5s	4-6s

Note: Benchmarks are approximate and depend on frame resolution, lighting, and number of simultaneous detections.

---

License

This project is provided as-is for educational and personal security use. Ensure you comply with local privacy laws when deploying camera systems.