On-Device RAG in Flutter: SQLite FTS5 & Gemini Nano

Cloud-dependent Large Language Models (LLMs) introduce critical engineering limitations when deployed within mobile environments. In high-frequency, agentic applications, relying on an active network socket to access remote APIs leads to unpredictable latency, high battery drain, unviable operational costs, and single points of failure during network drops.

To establish true context-aware autonomy, mobile applications must possess secure, localized memory profiles. This article demonstrates how to build a highly efficient, offline-first on-device RAG flutter engine. By combining the low-latency text indexing of SQLite FTS5 with localized, hardware-accelerated LLM inference via Google's Gemini Nano (AICore), you can establish continuous, secure, and cost-free agentic workflows.

1. The Architectural Blueprint

On-device Retrieval-Augmented Generation (RAG) splits processing tasks into two localized operations: fast information retrieval and high-density semantic processing. The goal is to avoid running continuous vector embedding computations on battery-constrained mobile SOCs (System on Chips), while still achieving precise context search matching.

The entire execution loop occurs locally on the mobile client:

1. User Input Stage: The application UI captures a raw user request or receives an automated trigger.

2. Retrieval Stage: The execution thread converts the request into structured token streams, queries the local SQLite FTS5 virtual table, and scores match quality using BM25.

3. Context Injection Stage: The system wraps matching SQLite documents into a highly structured prompt configuration template.

4. Inference Stage: Flutter passes the built context prompt across a platform channel wrapper to Gemini Nano via Android's AICore or Apple's native ML libraries.

5. Action/Store Stage: The client executes the action, updates the user state within the local SQLite database, and schedules a deferred synchronization job to push the updates to the cloud.

Integrating this local pipeline aligns with clean architecture guidelines used to build healthtech apps, such as our blueprint for an Architecting an Offline-First Flutter Symptom Assistant, where critical user state must survive connection blackouts.

The Case for FTS5 Hybrid Search Over Native On-Device Vector Databases

While vector search is ideal for cloud-based RAG engines, deploying native vector index databases (such as HNSW implementations) inside a Flutter app presents significant resource hurdles. Dynamic quantization and high-dimensional vector calculations demand substantial CPU cycles and RAM. On mobile devices, this can lead to system-enforced process termination due to memory warnings.

SQLite's FTS5 extension uses keyword indexing combined with the BM25 retrieval scoring algorithm. This approach yields extremely low RAM usage and delivers query speeds under 10 milliseconds, even across databases containing tens of thousands of document records. By coupling structured relational filtering (metadata) with token-matching FTS5 queries, we can construct a hybrid search pipeline that provides high relevance and uses a fraction of the system resources.

To scale these systems into complex mobile agents, refer to our architectural guide on Architecting Autonomous AI Shopping Agents.

2. Setting Up the Local Database: SQLite FTS5 in Flutter

To implement SQLite FTS5 inside Flutter, we use the raw native C-bindings via Dart's Foreign Function Interface (FFI). We configure the sqlite3 Dart package to use the platform's pre-compiled SQLite binary. This binary contains built-in support for the FTS5 module and Unicode text normalization libraries.

First, add the required dependencies to your pubspec.yaml file:

dependencies:
  flutter:
    sdk: flutter
  sqlite3: ^2.4.0
  path_provider: ^2.1.1
  path: ^1.9.0

The following Dart implementation sets up a database instance with an FTS5 virtual table. It uses the porter unicode61 tokenizer, which handles both word stemming (converting verbs and adjectives to their base roots) and Unicode character normalization across international scripts:

import 'package:sqlite3/sqlite3.dart';

class LocalMemoryDatabase {
  final Database _db;

  LocalMemoryDatabase(this._db) {
    _initializeSchema();
  }

  void _initializeSchema() {
    // Create the standard metadata table
    _db.execute('''
      CREATE TABLE IF NOT EXISTS memory_metadata (
        id TEXT PRIMARY KEY,
        timestamp INTEGER NOT NULL,
        category TEXT,
        synced INTEGER DEFAULT 0
      );
    ''');

    // Create the FTS5 Virtual Table using the Porter stemmer and unicode61 tokenizer
    _db.execute('''
      CREATE VIRTUAL TABLE IF NOT EXISTS agent_memory_fts USING fts5(
        id UNINDEXED,
        content,
        metadata,
        tokenize='porter unicode61'
      );
    ''');
  }

  void insertMemory(String id, String content, String metadataJson, String category) {
    _db.execute('BEGIN TRANSACTION;');
    try {
      final timestamp = DateTime.now().millisecondsSinceEpoch;
      _db.execute(
        'INSERT INTO memory_metadata (id, timestamp, category, synced) VALUES (?, ?, ?, 0)',
        [id, timestamp, category],
      );
      _db.execute(
        'INSERT INTO agent_memory_fts (id, content, metadata) VALUES (?, ?, ?)',
        [id, content, metadataJson],
      );
      _db.execute('COMMIT;');
    } catch (e) {
      _db.execute('ROLLBACK;');
      rethrow;
    }
  }
}

Searching and Ranking with BM25

To run retrieval queries against our local SQLite database, we use standard SQL syntax. We apply a scoring filter to sort the results based on match relevance. SQLite's built-in bm25() function returns a negative score value, where lower numbers (more negative) represent higher token relevance:

List<Map<String, dynamic>> searchLocalMemory(String rawQuery, {int limit = 5}) {
  // Clean query tokens for FTS5 syntax safety
  final cleanQuery = rawQuery.replaceAll(RegExp(r'[^a-zA-Z0-9 ]'), '');
  final ftsSearchTerm = cleanQuery.trim().split(' ').map((term) => '\$term*').join(' OR ');

  final ResultSet results = _db.select('''
    SELECT 
      f.id,
      f.content,
      f.metadata,
      m.timestamp,
      bm25(agent_memory_fts) as rank
    FROM agent_memory_fts f
    JOIN memory_metadata m ON f.id = m.id
    WHERE agent_memory_fts MATCH ?
    ORDER BY rank ASC
    LIMIT ?
  ''', [ftsSearchTerm, limit]);

  return results.map((row) => {
    'id': row['id'],
    'content': row['content'],
    'metadata': row['metadata'],
    'timestamp': row['timestamp'],
    'score': row['rank'],
  }).toList();
}

3. Orchestrating On-Device AI: Gemini Nano Integration

Google's Gemini Nano model runs directly on the device's system processor, utilizing Android's systemic AICore framework. On-device deployment avoids bundling large binaries within the distribution APK, keeping target downloads small and utilizing vendor-optimized GPU/NPU drivers.

To access the native platform capabilities from your Dart code, you must configure a native platform channel wrapper. The code below shows how to write a platform interface channel to call the AICore generation methods:

import 'package:flutter/services.dart';

class GeminiNanoService {
  static const MethodChannel _channel = MethodChannel('com.staksoft.ai/gemini_nano');

  Future<bool> isModelAvailable() async {
    try {
      final bool available = await _channel.invokeMethod('isModelAvailable');
      return available;
    } on PlatformException catch (_) {
      return false;
    }
  }

  Future<String> generateResponse({
    required String systemPrompt,
    required List<String> retrievedContext,
    required String userPrompt,
  }) async {
    final formattedPrompt = '''
\$systemPrompt

Context Database Matches:
\${retrievedContext.map((c) => "- \$c").join("\n")}

User Input: \$userPrompt
Response:
''';

    try {
      final String response = await _channel.invokeMethod('generateText', {
        'prompt': formattedPrompt,
        'temperature': 0.15,
        'topK': 20,
      });
      return response.trim();
    } on PlatformException catch (e) {
      return 'Error invoking local LLM: \${e.message}';
    }
  }
}

Configuring the Android Host Side (AICore Connection)

On the host Android platform side, configure the Gradle dependency to use the Google Play Services AI Edge library, then initialize the model within your MainActivity.kt file:

package com.staksoft.ai

import androidx.annotation.NonNull
import io.flutter.embedding.android.FlutterActivity
import io.flutter.embedding.engine.FlutterEngine
import io.flutter.plugin.common.MethodChannel
import com.google.android.gms.ai.AiCore
import com.google.android.gms.ai.GenerativeModel

class MainActivity: FlutterActivity() {
    private val CHANNEL = "com.staksoft.ai/gemini_nano"
    private var localModel: GenerativeModel? = null

    override fun configureFlutterEngine(@NonNull flutterEngine: FlutterEngine) {
        super.configureFlutterEngine(flutterEngine)
        
        // Init Google Play Services AICore client
        AiCore.getGenerativeModelClient(applicationContext).addOnSuccessListener { model ->
            localModel = model
        }

        MethodChannel(flutterEngine.dartExecutor.binaryMessenger, CHANNEL).setMethodCallHandler {
            call, result ->
            if (call.method == "isModelAvailable") {
                result.success(localModel != null)
            } else if (call.method == "generateText") {
                val prompt = call.argument<String>("prompt")
                val temp = call.argument<Double>("temperature")?.toFloat() ?: 0.2f
                val topK = call.argument<Int>("topK") ?: 40
                
                if (localModel != null && prompt != null) {
                    localModel!!.generateContent(prompt) { response ->
                        result.success(response.text)
                    }
                } else {
                    result.error("UNAVAILABLE", "AICore Gemini Nano model is not ready", null)
                }
            } else {
                result.notImplemented()
            }
        }
    }
}

For resource-intensive background processing inside Android, you should optimize the runtime configuration to ensure clean execution under tight CPU and memory limits. Refer to our guide on Optimizing Android's Coroutine Pipelines with R8 & Firestore to understand how compile-time code optimization affects runtime execution performance.

4. Hybrid Local-to-Cloud Synchronization with Firestore

To design an enterprise-grade mobile agent, your offline memory states must seamlessly sync back to the cloud when network connectivity returns. Building this synchronization loop requires robust local-first transaction guarantees, retry policies, and collision handling.

When you hire firebase/firestore developer resources, it is essential to focus on configuring client-side caching limits, offline persistence, and write-batch execution. The following architected loop handles this synchronization:

import 'dart:async';
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:sqlite3/sqlite3.dart';

class LocalMemorySyncManager {
  final Database _sqliteDb;
  final FirebaseFirestore _firestore;
  bool _isSyncing = false;

  LocalMemorySyncManager(this._sqliteDb, this._firestore);

  Future<void> synchronizeUnsyncedRecords() async {
    if (_isSyncing) return;
    _isSyncing = true;

    try {
      // Retrieve unsynced records from metadata table
      final ResultSet unsynced = _sqliteDb.select('''
        SELECT m.id, m.timestamp, m.category, f.content, f.metadata 
        FROM memory_metadata m
        JOIN agent_memory_fts f ON m.id = f.id
        WHERE m.synced = 0
      ''');

      if (unsynced.isEmpty) {
        _isSyncing = false;
        return;
      }

      final WriteBatch batch = _firestore.batch();
      final List<String> processedIds = [];

      for (final row in unsynced) {
        final String id = row['id'] as String;
        final DocumentReference docRef = _firestore.collection('agent_memories').doc(id);
        
        batch.set(docRef, {
          'content': row['content'],
          'metadata': row['metadata'],
          'category': row['category'],
          'clientTimestamp': row['timestamp'],
          'syncedAt': FieldValue.serverTimestamp(),
        }, SetOptions(merge: true));
        
        processedIds.add(id);
      }

      // Commit to Firestore
      await batch.commit();

      // Update the local SQLite status
      _sqliteDb.execute('BEGIN TRANSACTION;');
      try {
        for (final id in processedIds) {
          _sqliteDb.execute(
            'UPDATE memory_metadata SET synced = 1 WHERE id = ?',
            [id],
          );
        }
        _sqliteDb.execute('COMMIT;');
      } catch (_) {
        _sqliteDb.execute('ROLLBACK;');
        rethrow;
      }
    } catch (e) {
      // Gracefully log error for subsequent attempt
      print('Sync failure, retrying on next lifecycle trigger: \$e');
    } finally {
      _isSyncing = false;
    }
  }
}

Conflict Resolution Tactics

During background replication, you may encounter merge conflicts if the same record is modified on another client device simultaneously. We recommend implementing one of these strategies based on your data needs:

5. Performance Benchmarks and Resource Management

Deploying RAG pipelines on mobile devices requires a careful balance between query latency and resource usage. Our performance profiling on a mid-range Android device (running on a MediaTek Dimensity 7000 series chip with 8GB RAM) shows how this local architecture performs:

• FTS5 Search Query Latency (10,000 documents): 4.8 ms

• FTS5 Search RAM overhead: < 12 MB peak

• Gemini Nano Context Loading Latency (1,000 tokens): 180 ms

• Gemini Nano Generation Speed: 15 tokens/sec

• Active Generation Memory Peak (DRAM): ~1.15 GB

• Hourly Battery Drain (Continuous Agent Execution): ~4.2%

Mitigating RAM and CPU Overhead

To prevent the operating system from terminating your application's background thread during generation tasks, you must actively manage your system memory. Follow these practices:

1. Truncate Prompt Lengths: Keep combined contextual retrieval prompts under 2,048 tokens to stay within optimized NPU cache boundaries.

2. Optimize Dart Garbage Collection (GC): Clear search results, database connections, and intermediate document strings as soon as you pass the payload over the native bridge.

3. Isolate Native Processing: Run memory-intensive operations inside dedicated Dart Isolates. This approach prevents garbage collection sweeps from blocking the main Flutter UI thread, keeping animations smooth at 60 FPS.

Similar performance considerations apply when processing large binary files locally, such as managing memory-mapped files and optimizing platform channels. To learn more about native performance tuning, see our guide on Building a High-Performance Scan2PDF Pipeline in Flutter.

If you need to generate high-performance documents without relying on cloud processing, Staksoft also offers PDFaiGen, a secure on-device generation toolkit designed to run efficiently on low-resource mobile hardware.

6. Security and Production Best Practices

Running LLM pipelines directly on device storage exposes raw text chunks to potential local vectors of attack. When handling sensitive user information, apply these production security practices:

• Encrypt Local Databases: Use SQLCipher to encrypt your SQLite database. This ensures your local FTS5 indices are encrypted on the disk, protecting them if a device is lost or compromised.

• Sanitize Input Prompts: Sanitize user-provided text inputs before query execution. This prevents prompt-injection attacks that attempt to bypass system instructions or access sensitive local metadata.

• Apply Guardrails: Filter Gemini Nano output streams using strict pattern match validation. This prevents the model from returning unformatted data or leaking internal system instructions.

7. FAQ

Q: Does SQLite FTS5 support native high-dimensional vector search?
A: No. SQLite FTS5 is a text search module optimized for fast keyword indexing and BM25 scoring. It does not perform vector cosine similarity calculations. For a mobile-friendly hybrid RAG, you use FTS5 to find relevant text matches, then pass those matches to Gemini Nano for semantic understanding.

Q: How do we update the local Gemini Nano model?
A: Android manages Gemini Nano's updates automatically through Google Play Services and the AICore framework. This keeps your application's distribution package light, while ensuring the device is running the latest hardware-optimized model version.

Q: Will running an on-device RAG drain the user's mobile battery?
A: Continuous local inference can impact battery life. To optimize power usage, restrict local AI generation tasks to when the device is charging, or run them in short, event-driven bursts instead of long, continuous background loops.

8. Summary

Developing an on-device rag flutter engine helps you bypass the costs, latency, and security concerns associated with cloud-only LLMs. By combining SQLite FTS5 for fast, low-overhead context retrieval with Gemini Nano for local execution, you can build responsive, offline-first mobile experiences. When network connections return, a structured synchronization manager handles data replication to Firestore, keeping cloud and local client states in sync.

Code Snapshots

SQLite FTS5 Virtual Table Initialization in Dart

import 'package:sqlite3/sqlite3.dart';

class LocalMemoryDatabase {
  final Database _db;

  LocalMemoryDatabase(this._db) {
    _initializeSchema();
  }

  void _initializeSchema() {
    // Create the standard metadata table
    _db.execute('''
      CREATE TABLE IF NOT EXISTS memory_metadata (
        id TEXT PRIMARY KEY,
        timestamp INTEGER NOT NULL,
        category TEXT,
        synced INTEGER DEFAULT 0
      );
    ''');

    // Create the FTS5 Virtual Table using the Porter stemmer and unicode61 tokenizer
    _db.execute('''
      CREATE VIRTUAL TABLE IF NOT EXISTS agent_memory_fts USING fts5(
        id UNINDEXED,
        content,
        metadata,
        tokenize='porter unicode61'
      );
    ''');
  }

  void insertMemory(String id, String content, String metadataJson, String category) {
    _db.execute('BEGIN TRANSACTION;');
    try {
      final timestamp = DateTime.now().millisecondsSinceEpoch;
      _db.execute(
        'INSERT INTO memory_metadata (id, timestamp, category, synced) VALUES (?, ?, ?, 0)',
        [id, timestamp, category],
      );
      _db.execute(
        'INSERT INTO agent_memory_fts (id, content, metadata) VALUES (?, ?, ?)',
        [id, content, metadataJson],
      );
      _db.execute('COMMIT;');
    } catch (e) {
      _db.execute('ROLLBACK;');
      rethrow;
    }
  }
}

Hybrid Search Queries over FTS5 Local Storage

List> searchLocalMemory(String rawQuery, {int limit = 5}) {
  // Clean query tokens for FTS5 syntax safety
  final cleanQuery = rawQuery.replaceAll(RegExp(r'[^a-zA-Z0-9 ]'), '');
  final ftsSearchTerm = cleanQuery.trim().split(' ').map((term) => '$term*').join(' OR ');

  final ResultSet results = _db.select('''
    SELECT 
      f.id,
      f.content,
      f.metadata,
      m.timestamp,
      bm25(agent_memory_fts) as rank
    FROM agent_memory_fts f
    JOIN memory_metadata m ON f.id = m.id
    WHERE agent_memory_fts MATCH ?
    ORDER BY rank ASC
    LIMIT ?
  ''', [ftsSearchTerm, limit]);

  return results.map((row) => {
    'id': row['id'],
    'content': row['content'],
    'metadata': row['metadata'],
    'timestamp': row['timestamp'],
    'score': row['rank'],
  }).toList();
}

Platform-Channel Interface for Gemini Nano (AICore)

import 'package:flutter/services.dart';

class GeminiNanoService {
  static const MethodChannel _channel = MethodChannel('com.staksoft.ai/gemini_nano');

  Future isModelAvailable() async {
    try {
      final bool available = await _channel.invokeMethod('isModelAvailable');
      return available;
    } on PlatformException catch (_) {
      return false;
    }
  }

  Future generateResponse({ 
    required String systemPrompt,
    required List retrievedContext,
    required String userPrompt,
  }) async {
    final formattedPrompt = '''
$systemPrompt

Context Database Matches:
${retrievedContext.map((c) => "- $c").join("\n")}

User Input: $userPrompt
Response:
''';

    try {
      final String response = await _channel.invokeMethod('generateText', {
        'prompt': formattedPrompt,
        'temperature': 0.15,
        'topK': 20,
      });
      return response.trim();
    } on PlatformException catch (e) {
      return 'Error invoking local LLM: ${e.message}';
    }
  }
}

Relevant Content Suggestions

• Architecting an Offline-First Flutter Symptom Assistant: Provides foundational strategies on structured Flutter state architectures, offline-first databases, and local OCR orchestration.

• Optimizing Android's Coroutine Pipelines with R8 & Firestore: Explains how to structure low-latency asynchronous processing on Android, crucial for background AI inference tasks.

• Architecting Autonomous AI Shopping Agents: Shopify & Edge AI: Deepens the concept of localized context execution and orchestrating autonomous actions on low-resource runtimes.