YapHub

Appearance

Yappa Knowledge Hub - Slack Bot Architecture ​

Overview ​

The Yappa Knowledge Hub is a Slack bot that integrates with Notion to manage knowledge items, categories, and digests. It uses a hybrid architecture with Notion as the source of truth and a local SQLite database as a performance cache.

System Architecture ​


   Slack Users     Node.js Bot       Symfony API
                         (Port 3000)             (Port 8000)




                                                      SQLite Cache
                                                      + Notion API

Data Sources ​

1. Notion Databases (Source of Truth) ​

  • Knowledge Database: 306e292a-15d5-8004-a8cb-c222dcd48bb2
    • Stores all knowledge items with title, content, tags, categories, target groups
  • Categories Database: 306e292a-15d5-805d-ae13-e64bed8519c5
    • Stores categories (lists) with name, description, icon, settings
  • Digests Database: 306e292a-15d5-80d7-a0f6-fe8421baff10
    • Stores generated digests for periodic summaries

2. SQLite Local Database ​

  • Location: https://github.com/undead2146/KnowledgeHub/blob/main/backend/var/data.db
  • Purpose: Performance cache to reduce Notion API calls
  • Entities: Knowledge, Category
  • Sync Strategy:
    • Auto-sync every 5 minutes when data is accessed
    • Manual sync via API endpoint
    • Timestamp-based comparison to detect changes

Data Flow ​

Write Flow (Creating Knowledge) ​

  1. User interacts with Slack (modal, shortcut, reaction)
  2. Node.js handler processes input
  3. Calls src/services/knowledge.js addKnowledge()
  4. HTTP POST to http://localhost:8000/api/knowledge
  5. Symfony KnowledgeController receives request
  6. NotionKnowledgeService creates page in Notion
  7. Response returns with Notion ID and URL
  8. Optional: AI summary generation

Read Flow (Listing Knowledge) ​

  1. Slack bot calls listKnowledge() or listCategories()
  2. HTTP GET to Symfony API
  3. Symfony checks if auto-sync needed (5-minute TTL)
  4. If sync needed: Fetch from Notion Update SQLite
  5. Return data from SQLite (faster)
  6. Node.js formats data for Slack UI

Slack Bot Components ​

Entry Point ​

File: src/app.js

  • Initializes Slack Bolt app with Socket Mode
  • Registers all handlers
  • Starts Express API server for webhooks
  • Initializes home refresh service

Handlers ​

1. Commands Handler (src/handlers/commands.js) ​

Handles slash commands:

  • /knowledge - Main command dispatcher
  • /knowledge add - Open add modal
  • /knowledge search <query> - Search knowledge
  • /knowledge list - Dashboard with filters
  • /knowledge help - Show help

2. Events Handler (src/handlers/events.js) ​

Handles Slack events:

  • app_home_opened - Render home tab
  • message - Process messages
  • file_shared - Handle file attachments

3. Interactions Handler (src/handlers/interactions.js) ​

Handles all interactive components:

  • View Submissions: Modal form submissions
  • Button Actions: All button clicks
  • Action Patterns:
    • home_* - Home tab actions
    • browse_list_{id} - View category items
    • view_item_{id} - View single item
    • edit_list_{id} - Edit category
    • delete_list_{id} - Delete category
    • manage_lists_*_page - Pagination

4. Shortcuts Handler (src/handlers/shortcuts.js) ​

Handles Slack shortcuts:

  • save_to_knowledge / add_to_hub - Message shortcuts (right-click)
  • quick_add_knowledge - Global shortcut (lightning bolt menu)

5. Home Handler (src/handlers/home.js) ​

Manages the App Home tab:

  • Renders main dashboard
  • Shows stats, categories, recent items
  • Handles pagination
  • Manages browse list modal

6. Search Modal Handler (src/handlers/searchModal.js) ​

Dedicated search interface:

  • Advanced filtering options
  • Category and target group filters

Services ​

Node.js Services (src/services/) ​

knowledge.js

  • addKnowledge() - Create new knowledge item
  • listKnowledge() - List all knowledge items
  • getKnowledge(id) - Get single item
  • updateKnowledge(id, data) - Update item
  • deleteKnowledge(id) - Delete item
  • searchKnowledge(query) - Search items

categories.js

  • listCategories() - List all categories
  • getCategory(id) - Get single category
  • addCategory(data) - Create category
  • updateCategory(id, data) - Update category
  • deleteCategory(id) - Delete category

homeRefresh.js

  • initHomeRefreshService(app) - Initialize with Slack app instance
  • refreshAllHomes() - Refresh all active home views
  • refreshUserHome(userId) - Refresh specific user's home
  • Tracks active home views to minimize API calls

viewTracker.js

  • trackView(userId, viewType, metadata) - Track open modal
  • getTrackedView(userId) - Get user's current view
  • clearTrackedView(userId) - Clear tracked view
  • startPeriodicCleanup() - Clean up stale views every 5 minutes

aiSummary.js

  • generateSummary(content) - Generate AI summary (currently disabled)
  • UI components for summary display

urlScraper.js

  • scrapeUrl(url) - Extract metadata from URLs
  • Returns title, description, image

Views and Modals ​

App Home View ​

Purpose: Main dashboard showing stats, categories, recent items

Components:

  • Welcome header with quick action buttons
  • Stats display (total items, total lists)
  • Paginated category list (10 per page)
  • Recent items section (10 most recent)
  • Tips and help text

Pagination: Handles large category lists with prev/next buttons

Modals ​

1. Add Knowledge Modal (add_knowledge_modal) ​

Triggered by: /knowledge add, home button, global shortcut

Fields:

  • Title (required)
  • Content (required)
  • Tags (optional, comma-separated)
  • Category (select from active categories)
  • Target Groups (multi-select)

Flow:

  1. User fills form
  2. Submit handleViewSubmission()
  3. POST to /api/knowledge
  4. Notion page created
  5. Home views refreshed
  6. Success message sent

2. Save Message Modal (save_message_modal) ​

Triggered by: Message shortcut (right-click on message)

Pre-filled:

  • Content from message text
  • Source message metadata (user, channel, timestamp)

Additional Fields:

  • Title (required)
  • Tags (optional)
  • Category (select)
  • Target Groups (multi-select)

3. Quick Add Modal (quick_add_modal) ​

Triggered by: Global shortcut (lightning bolt menu)

Fields:

  • Title (required)
  • Content or URL (required)
  • Tags (optional)
  • Category (select)
  • Target Groups (multi-select)

Special: If URL provided, scrapes metadata automatically

4. Edit Knowledge Modal (edit_knowledge_modal_{id}) ​

Triggered by: Edit button on knowledge item

Pre-filled: All existing data from Notion

Flow:

  1. Fetch current data from Notion
  2. Pre-fill form
  3. User edits
  4. Submit PUT to /api/knowledge/{id}
  5. Updates both Notion and SQLite
  6. Refreshes views

5. Manage Lists Modal (manage_lists_modal) ​

Triggered by: "Beheer Lijsten" button on home

Features:

  • Shows all categories (active and inactive)
  • Edit/Delete buttons per category
  • Create new category button
  • Pagination (10 per page)
  • Shows item count per category

Actions:

  • edit_list_{id} - Opens edit category modal
  • delete_list_{id} - Confirms and deletes category
  • manage_lists_create - Opens create category modal
  • manage_lists_prev_page / manage_lists_next_page - Pagination

6. Browse List Modal (browse_list_modal) ​

Triggered by: Clicking category name on home

Features:

  • Shows all items in selected category
  • Item preview with title, tags, target groups
  • "Bekijk" button per item
  • Pagination if many items

Flow:

  1. User clicks category
  2. Fetch items by category from API
  3. Display in modal
  4. Click "Bekijk" Opens view item modal

7. Add/Edit Category Modals ​

Fields:

  • Name (required)
  • Description (optional)
  • Icon (emoji picker)
  • Default Target Groups (multi-select)
  • Is Active (checkbox)
  • Sort Order (number)

Icon Handling:

  • Slack emoji format (:smile:) Unicode emoji ()
  • Stored as Unicode in Notion
  • Displayed as emoji in Slack

8. Search Modal ​

Triggered by: /knowledge search or search button

Features:

  • Search query input
  • Category filter (multi-select)
  • Target group filter (multi-select)
  • Results with preview
  • Click to view full item

Backend (Symfony) Components ​

Controllers ​

KnowledgeController (src/Controller/KnowledgeController.php) ​

Endpoints:

  • GET /api/knowledge - List all knowledge items
  • GET /api/knowledge/{id} - Get single item
  • POST /api/knowledge - Create new item
  • PUT /api/knowledge/{id} - Update item
  • DELETE /api/knowledge/{id} - Delete item
  • GET /api/knowledge/search - Search items

CategoryController (src/Controller/CategoryController.php) ​

Endpoints:

  • GET /api/categories - List all categories (with auto-sync)
  • GET /api/categories/{id} - Get single category
  • POST /api/categories - Create new category
  • PUT /api/categories/{id} - Update category
  • DELETE /api/categories/{id} - Delete category
  • POST /api/categories/sync - Manual sync from Notion

Auto-Sync Logic:

php
// Checks if last sync was > 5 minutes ago
if ($lastSync === null || (time() - $lastSync->getTimestamp()) > 300) {
    $this->syncFromNotion();
}

Services ​

NotionClient (src/Service/NotionClient.php) ​

Purpose: HTTP client for Notion API with retry logic

Methods:

  • queryDatabase($databaseId, $filter, $sorts) - Query database (max 100 results)
  • queryDatabaseWithPagination($databaseId, $filter, $sorts) - Query all results
  • getPage($pageId) - Get single page
  • createPage($databaseId, $properties) - Create page
  • updatePage($pageId, $properties) - Update page
  • deletePage($pageId) - Archive page

Features:

  • Automatic retry on rate limits (429)
  • Exponential backoff
  • Error logging

NotionKnowledgeService (src/Service/NotionKnowledgeService.php) ​

Purpose: CRUD operations for knowledge items

Methods:

  • listAll() - List all items (uses pagination)
  • getById($id) - Get single item
  • create($data) - Create item in Notion
  • update($id, $data) - Update item
  • delete($id) - Archive item
  • search($query) - Search items
  • queryByCategory($categoryId) - Get items by category

Property Mapping:

  • Title title (title property)
  • Content content (rich_text property)
  • Tags tags (multi_select property)
  • Category category (relation property)
  • Target Groups target_groups (multi_select property)

NotionCategoryService (src/Service/NotionCategoryService.php) ​

Purpose: CRUD operations for categories

Methods:

  • listAll() - List all categories
  • getById($id) - Get single category
  • create($data) - Create category
  • update($id, $data) - Update category
  • delete($id) - Archive category

** KNOWN ISSUE**: listAll() does NOT use pagination, causing count discrepancies!

NotionSyncService (src/Service/NotionSyncService.php) ​

Purpose: Bidirectional sync between SQLite and Notion

Methods:

  • syncFromNotion() - Fetch from Notion Update SQLite
  • syncToNotion() - Push from SQLite Update Notion
  • syncCategory($notionCategory) - Sync single category
  • syncKnowledge($notionKnowledge) - Sync single knowledge item

Sync Strategy:

  • Compare notionLastEditedAt with lastSyncedAt
  • Only update if Notion is newer
  • Batch operations for efficiency

NotionPropertyMapper (src/Service/NotionPropertyMapper.php) ​

Purpose: Map Notion properties to/from arrays

Methods:

  • mapPropertiesToArray($properties) - Notion Array
  • mapArrayToProperties($data) - Array Notion
  • extractTitle($property) - Extract title text
  • extractRichText($property) - Extract rich text
  • extractMultiSelect($property) - Extract multi-select values
  • extractRelation($property) - Extract relation IDs

Caching & Synchronization ​

Caching Strategy ​

  • SQLite as Cache: Local database caches Notion data
  • TTL: 5-minute cache for categories
  • Lazy Loading: Only syncs when data is accessed
  • Timestamp Comparison: Only updates if Notion was edited after last sync

Synchronization Mechanisms ​

Category Sync ​

Trigger: Every 5 minutes when categories are accessed

Process:

  1. Check lastSyncedAt timestamp
  2. If > 5 minutes, fetch from Notion
  3. Compare notionLastEditedAt with lastSyncedAt
  4. Update SQLite if Notion is newer
  5. Update fields: name, description, icon, targetGroups, isActive, sortOrder

Knowledge Sync ​

Trigger: Manual or on-demand

Process:

  1. Fetch all items from Notion (with pagination)
  2. Compare with SQLite
  3. Update/Create/Delete as needed
  4. Update timestamps

Home Refresh Service ​

Purpose: Keep Slack home views in sync with data changes

How it works:

  1. Node.js bot tracks active home views
  2. After data changes, Symfony calls Node.js API
  3. Node.js refreshes all tracked home views
  4. Users see updated data immediately

Endpoints:

  • POST /api/slack/refresh-homes - Refresh all homes
  • POST /api/slack/refresh-home - Refresh specific user

Known Issues & Solutions ​

Issue 1: "75 items vs 18 items" Discrepancy ​

Root Cause: NotionCategoryService::listAll() does NOT use pagination

Location: backend/src/Service/NotionCategoryService.php:118-122

Current Code:

php
public function listAll(): array {
    $response = $this->client->queryDatabase($this->databaseId, []);
    return $response['results'] ?? [];
}

Problem: queryDatabase() returns max 100 items per page, but doesn't handle pagination

Solution: Use queryDatabaseWithPagination() like NotionKnowledgeService does:

php
public function listAll(): array {
    $response = $this->client->queryDatabaseWithPagination($this->databaseId, [], []);
    return $response['results'] ?? [];
}

Issue 2: Connection Refused Errors ​

Symptom: ECONNREFUSED 127.0.0.1:8000

Root Cause: Symfony backend not running when Node.js bot starts

Solution:

  1. Start Symfony backend first: php -S localhost:8000 -t public/
  2. Then start Node.js bot: npm start
  3. Or use process manager (PM2, systemd) to ensure both run

Issue 3: Stale Cache ​

Symptom: Slack shows old data even after Notion updates

Root Cause: 5-minute cache TTL

Solutions:

  1. Manual sync: POST /api/categories/sync
  2. Reduce TTL in CategoryController
  3. Implement webhook from Notion (not currently supported by Notion)

Issue 4: Inconsistent View State ​

Symptom: Modals show wrong data or don't update

Root Cause: View tracker not clearing properly

Solution: viewTracker.js runs periodic cleanup every 5 minutes

Configuration ​

Environment Variables ​

Node.js (.env):

env
SLACK_BOT_TOKEN=xoxb-...
SLACK_SIGNING_SECRET=...
SLACK_APP_TOKEN=xapp-...
PORT=3000
API_PORT=3001
BACKEND_URL=http://localhost:8000

Symfony (backend/.env):

env
NOTION_API_KEY=secret_...
NOTION_DATABASE_KNOWLEDGE=306e292a-15d5-8004-a8cb-c222dcd48bb2
NOTION_DATABASE_CATEGORIES=306e292a-15d5-805d-ae13-e64bed8519c5
NOTION_DATABASE_DIGESTS=306e292a-15d5-80d7-a0f6-fe8421baff10
DATABASE_URL=sqlite:///%kernel.project_dir%/var/data.db
SLACK_BOT_URL=http://localhost:3001

API Endpoints ​

Node.js API (Port 3001):

  • POST /api/slack/refresh-homes - Refresh all home pages
  • POST /api/slack/refresh-home - Refresh specific user home
  • GET /health - Health check

Symfony API (Port 8000):

  • GET /api/knowledge - List knowledge items
  • POST /api/knowledge - Create knowledge item
  • GET /api/categories - List categories (with auto-sync)
  • POST /api/categories/sync - Manual sync

Deployment ​

Starting the Application ​

  1. Start Symfony Backend:

    bash
    cd backend
php -S localhost:8000 -t public/

  2. Start Node.js Bot:

    bash
    npm start

  3. Verify:

    • Symfony: curl http://localhost:8000/
    • Node.js: curl http://localhost:3001/health
    • Slack: Open app home tab

Production Considerations ​

  1. Process Management: Use PM2 or systemd to keep services running
  2. Logging: Configure proper log rotation
  3. Monitoring: Set up health checks and alerts
  4. Database Backups: Regular backups of SQLite database
  5. Error Handling: Implement proper error recovery
  6. Rate Limiting: Respect Notion API rate limits (3 requests/second)

Troubleshooting ​

Bot Not Responding ​

  1. Check if Node.js process is running: ps aux | grep node
  2. Check logs: tail -f logs/slack-bot.log
  3. Verify Slack tokens in .env
  4. Check Socket Mode connection

Data Not Syncing ​

  1. Check if Symfony backend is running: curl http://localhost:8000/
  2. Check last sync time: GET /api/categories (look for lastSyncedAt)
  3. Manual sync: POST /api/categories/sync
  4. Check Notion API key validity

Modals Not Opening ​

  1. Check view tracker: Look for [View Tracker] logs
  2. Clear stale views: Restart bot
  3. Check modal payload size (Slack has 3000 character limit)
  4. Verify category metadata size

Performance Issues ​

  1. Check SQLite database size
  2. Verify pagination is working
  3. Monitor Notion API rate limits
  4. Consider increasing cache TTL

Future Improvements ​

  1. Fix Pagination: Update NotionCategoryService::listAll() to use pagination
  2. Real-time Sync: Implement webhook from Notion (when available)
  3. Better Caching: Implement Redis for distributed caching
  4. Search Optimization: Add full-text search to SQLite
  5. AI Summaries: Re-enable OpenAI integration for automatic summaries
  6. Analytics: Track usage metrics and popular categories
  7. Bulk Operations: Add bulk edit/delete capabilities
  8. Export: Add export to PDF/Markdown
  9. Permissions: Add role-based access control
  10. Notifications: Add digest notifications to channels