Accessing Project Data from Code Nodes

Code nodes can securely access your project's library files and databases, enabling powerful data-driven workflows. Read files, query databases, generate reports, and persist results—all from within your JavaScript or Python code.

What is Code Node Data Access?

Code node data access enables your custom code to interact with Intellectible's built-in storage systems:

Library Access - List, download, upload, and manage files in your project library
Database Access - Query, insert, update, and delete data in your project databases

This unlocks workflows that combine custom code with persistent data storage, without requiring external databases or file storage services.

Built-In SDK

The LibraryClient and DatabaseClient SDKs are pre-installed in all code execution environments:

JavaScript:

const { LibraryClient, DatabaseClient } = require('@intellectible/execution-sdk');

const library = new LibraryClient();
const database = new DatabaseClient();

Python:

from intellectible_execution import LibraryClient, DatabaseClient

library = LibraryClient()
database = DatabaseClient()

No installation, no configuration—just import and use.

When to Use Data Access

Library Access Use Cases

Use Case	Description	Example
File Processing	Download and transform files	Convert PDF to text, resize images
Report Generation	Create and upload reports	Generate CSV/PDF from workflow results
Data Import	Process uploaded files	Parse CSV, extract data from documents
File Management	Organize library programmatically	Archive old files, rename/move files
Content Analysis	Analyze file contents	Sentiment analysis, keyword extraction

Database Access Use Cases

Use Case	Description	Example
Execution Logging	Track workflow runs	Log timestamps, inputs, results
Data Aggregation	Collect results over time	Build metrics dashboard
Cache/Lookup	Store reusable data	API response cache, reference tables
Audit Trail	Record changes and events	User actions, system events
State Management	Persist workflow state	Resume interrupted workflows

Common Patterns

Pattern 1: Process Files from Library

Download files, transform them, and upload results back to the library.

JavaScript:

const { LibraryClient } = require('@intellectible/execution-sdk');
const fs = require('fs');

const library = new LibraryClient();

// Read inputs
const inputs = JSON.parse(fs.readFileSync('inputs.json', 'utf8'));

// List all CSV files
const files = await library.listFiles();
const csvFiles = files.filter(f => f.type === 'text/csv');

console.log(`Found ${csvFiles.length} CSV files`);

let result;

// Download and process first CSV
if (csvFiles.length > 0) {
  const downloadUrl = await library.createDownloadUrl(csvFiles[0].id);
  const response = await fetch(downloadUrl);
  const csvData = await response.text();

  // Process CSV data (parse, transform, aggregate)
  const rows = csvData.split('\n');
  const summary = `Processed ${rows.length} rows from ${csvFiles[0].name}`;

  // Upload summary as new file (single call handles everything)
  await library.uploadFile('summary.txt', 'txt', summary);

  result = { processed: csvFiles[0].name, summary };
} else {
  result = { message: 'No CSV files found' };
}

// Write output
fs.writeFileSync('output.json', JSON.stringify(result, null, 2));

Python:

import json
import requests
from intellectible_execution import LibraryClient

library = LibraryClient()

# Read inputs
with open('inputs.json', 'r') as f:
    inputs = json.load(f)

# List all CSV files
files = library.list_files()
csv_files = [f for f in files if f['type'] == 'text/csv']

print(f'Found {len(csv_files)} CSV files')

# Download and process first CSV
if csv_files:
    download_url = library.create_download_url(csv_files[0]['id'])
    response = requests.get(download_url)
    csv_data = response.text

    # Process CSV data
    rows = csv_data.split('\n')
    summary = f'Processed {len(rows)} rows from {csv_files[0]["name"]}'

    # Upload summary (single call handles everything)
    library.upload_file('summary.txt', 'txt', summary.encode('utf-8'))

    result = {'processed': csv_files[0]['name'], 'summary': summary}
else:
    result = {'message': 'No CSV files found'}

# Write output
with open('output.json', 'w') as f:
    json.dump(result, f)

Pattern 2: Log Workflow Execution

Create an execution log table and record each workflow run.

JavaScript:

const { DatabaseClient } = require('@intellectible/execution-sdk');
const fs = require('fs');

const database = new DatabaseClient();

// Read inputs
const inputs = JSON.parse(fs.readFileSync('inputs.json', 'utf8'));

// Use an existing database
const databases = await database.listDatabases();
const logDb = databases[0];

// Get or create table
const tables = await database.listTables(logDb.id);

if (!tables.includes('executions')) {
  await database.createTable(logDb.id, 'executions');
  await database.createColumn(logDb.id, 'executions', 'workflow_name', 'text');
  await database.createColumn(logDb.id, 'executions', 'status', 'text');
  await database.createColumn(logDb.id, 'executions', 'executed_at', 'text');
  await database.createColumn(logDb.id, 'executions', 'duration_ms', 'number');
  await database.createColumn(logDb.id, 'executions', 'result', 'jsonb');
}

// Log current execution
const executionStart = Date.now();

// ... perform workflow logic ...
const workflowResult = { success: true, items_processed: inputs.count };

const duration = Date.now() - executionStart;

await database.insertRows(logDb.id, 'executions', [{
  workflow_name: inputs.workflowName || 'unnamed',
  status: 'success',
  executed_at: new Date().toISOString(),
  duration_ms: duration,
  result: JSON.stringify(workflowResult)
}]);

// Query recent executions
const recent = await database.query(logDb.id, 'executions',
  'SELECT * FROM "executions" ORDER BY "executed_at" DESC LIMIT 5'
);

const output = {
  logged: true,
  recentExecutions: recent.length,
  thisExecution: { duration, result: workflowResult }
};

// Write output
fs.writeFileSync('output.json', JSON.stringify(output, null, 2));

Python:

from intellectible_execution import DatabaseClient
from datetime import datetime
import json
import time

database = DatabaseClient()

# Read inputs
with open('inputs.json', 'r') as f:
    inputs = json.load(f)

# Use an existing database
databases = database.list_databases()
log_db = databases[0]

# Get or create table
tables = database.list_tables(log_db['id'])

if 'executions' not in tables:
    database.create_table(log_db['id'], 'executions')
    database.create_column(log_db['id'], 'executions', 'workflow_name', 'text')
    database.create_column(log_db['id'], 'executions', 'status', 'text')
    database.create_column(log_db['id'], 'executions', 'executed_at', 'text')
    database.create_column(log_db['id'], 'executions', 'duration_ms', 'number')
    database.create_column(log_db['id'], 'executions', 'result', 'jsonb')

# Log current execution
execution_start = time.time()

# ... perform workflow logic ...
workflow_result = {'success': True, 'items_processed': inputs.get('count', 0)}

duration = int((time.time() - execution_start) * 1000)

database.insert_rows(log_db['id'], 'executions', [{
    'workflow_name': inputs.get('workflowName', 'unnamed'),
    'status': 'success',
    'executed_at': datetime.now().isoformat(),
    'duration_ms': duration,
    'result': json.dumps(workflow_result)
}])

# Query recent executions
recent = database.query(log_db['id'], 'executions',
    'SELECT * FROM "executions" ORDER BY "executed_at" DESC LIMIT 5'
)

output = {
    'logged': True,
    'recent_executions': len(recent),
    'this_execution': {'duration': duration, 'result': workflow_result}
}

# Write output
with open('output.json', 'w') as f:
    json.dump(output, f)

Pattern 3: Build Incremental Data Pipeline

Read from library, process data, store results in database.

JavaScript:

const { LibraryClient, DatabaseClient } = require('@intellectible/execution-sdk');
const fs = require('fs');

const library = new LibraryClient();
const database = new DatabaseClient();

// Setup database
const databases = await database.listDatabases();
const pipelineDb = databases[0];
const dbId = pipelineDb.id;

const tables = await database.listTables(dbId);
if (!tables.includes('processed_files')) {
  await database.createTable(dbId, 'processed_files');
  await database.createColumn(dbId, 'processed_files', 'file_name', 'text', { colIsUnique: true });
  await database.createColumn(dbId, 'processed_files', 'processed_at', 'text');
  await database.createColumn(dbId, 'processed_files', 'record_count', 'number');
}

// Get already processed files
const processed = await database.query(pipelineDb.id, 'processed_files',
  'SELECT * FROM "processed_files"'
);
const processedNames = new Set(processed.map(r => r.file_name));

// Find unprocessed CSV files
const files = await library.listFiles();
const unprocessedCsvs = files.filter(f =>
  f.type === 'text/csv' && !processedNames.has(f.name)
);

console.log(`Found ${unprocessedCsvs.length} new files to process`);

// Process each file
for (const file of unprocessedCsvs) {
  const downloadUrl = await library.createDownloadUrl(file.id);
  const response = await fetch(downloadUrl);
  const csvData = await response.text();

  const rowCount = csvData.split('\n').length - 1; // Exclude header

  // Mark as processed
  await database.insertRows(pipelineDb.id, 'processed_files', [{
    file_name: file.name,
    processed_at: new Date().toISOString(),
    record_count: rowCount
  }]);

  console.log(`Processed ${file.name}: ${rowCount} records`);
}

const output = {
  newFiles: unprocessedCsvs.length,
  totalProcessed: processed.length + unprocessedCsvs.length
};

// Write output
fs.writeFileSync('output.json', JSON.stringify(output, null, 2));

Pattern 4: Bulk CSV Import to Database

Upload a CSV file to the library, then import it directly into a database table.

JavaScript:

const { LibraryClient, DatabaseClient } = require('@intellectible/execution-sdk');
const fs = require('fs');

const library = new LibraryClient();
const database = new DatabaseClient();

// Generate CSV data
const csvData = [
  'name,email,age',
  'Alice,alice@example.com,30',
  'Bob,bob@example.com,25',
  'Charlie,charlie@example.com,35'
].join('\n');

// Upload to library (single call handles everything)
const docId = await library.uploadFile('users.csv', 'csv', csvData);

// Use an existing database and create table
const databases = await database.listDatabases();
const dbId = databases[0].id;
await database.createTable(dbId, 'users');
await database.createColumn(dbId, 'users', 'name', 'text');
await database.createColumn(dbId, 'users', 'email', 'text', {
  colIsUnique: true
});
await database.createColumn(dbId, 'users', 'age', 'number');

// Import CSV into table
const importResult = await database.uploadCSV(dbId, 'users', docId);

const output = {
  fileId: docId,
  rowsImported: importResult.rowsInserted,
  errors: importResult.errors
};

// Write output
fs.writeFileSync('output.json', JSON.stringify(output, null, 2));

uploadCSV keeps the synchronous SDK behavior for code nodes. For large CSV files, use the database table panel's Import button instead; it imports in background batches and shows progress on the table page.

Security Model

Project Isolation

Code nodes can only access data from their own project:

Library files - Only files in the current project's library
Databases - Only databases created in the current project
No cross-project access - Cannot read or write to other projects

This isolation is enforced by the execution token, which is cryptographically bound to the project ID.

Automatic Authentication

The SDK handles all authentication automatically:

No API keys in code - Token provided by Intellectible infrastructure
No credentials to manage - Authentication is transparent
Token lifecycle - Created when execution starts, destroyed when it ends
Short-lived - 10-minute expiration limits exposure

Network Security

Code execution containers have restricted network access:

✅ Allowed - Project library and database APIs
✅ Allowed - External HTTP/HTTPS APIs

Best Practices

Never log tokens - SDK abstracts token handling
Handle errors gracefully - Database/library operations can fail
Validate data - Don't trust user inputs blindly
Limit query sizes - Use pagination for large result sets
Clean up resources - Delete temporary files/tables when done

Performance Considerations

Library Operations

Download time - Large files take longer to download
Upload time - Large files take longer to upload
Processing time - Factor in time to parse/transform file contents

Tip: For files >10MB, consider streaming or chunked processing.

Database Operations

Use LIMIT - Always add a LIMIT clause to SQL queries on large tables
Indexes - Use colIsIndexed: true on frequently queried columns
Bulk imports - Use insertRows with multiple rows or uploadCSV for bulk imports
JSON columns - Avoid very large JSON objects (>1MB)

Execution Timeout

Code nodes have a 10-minute execution timeout. Plan accordingly:

Budget time for library downloads (~5-30 seconds for large files)
Budget time for database queries (~100ms-5s depending on data size)
Leave buffer time for your processing logic

Limitations

Execution timeout - 10 minutes total (including all SDK operations)
No persistent state - Containers are destroyed after execution

Troubleshooting

Common Errors

"Failed to list files: Unauthorized"

Cause: Execution token expired or invalid
Solution: This should not happen during normal execution; contact support

"Failed to query table: Table not found"

Cause: Table doesn't exist or wrong table ID
Solution: Use listTables() to verify table names

"Query timeout"

Cause: Query taking too long (>30 seconds)
Solution: Add pagination, reduce result set size, or add indexes

"File download failed: 404"

Cause: File ID doesn't exist or was deleted
Solution: Use listFiles() to get current file IDs

Debugging Tips

Log operations - Use console.log to trace SDK calls
Check return values - Verify IDs returned from create operations
Test incrementally - Test library and database access separately
Verify data types - Ensure column types match inserted data
Handle errors - Wrap SDK calls in try-catch blocks

Next Steps

Ready to use code node data access in your workflows?

API Documentation:

Related Guides:

What is Code Node Data Access?​

Built-In SDK​

When to Use Data Access​

Library Access Use Cases​

Database Access Use Cases​

Common Patterns​

Pattern 1: Process Files from Library​

Pattern 2: Log Workflow Execution​

Pattern 3: Build Incremental Data Pipeline​

Pattern 4: Bulk CSV Import to Database​

Security Model​

Project Isolation​

Automatic Authentication​

Network Security​

Best Practices​

Performance Considerations​

Library Operations​

Database Operations​

Execution Timeout​

Limitations​

Limitations​

Troubleshooting​

Common Errors​

Debugging Tips​

Next Steps​

What is Code Node Data Access?

Built-In SDK

When to Use Data Access

Library Access Use Cases

Database Access Use Cases

Common Patterns

Pattern 1: Process Files from Library

Pattern 2: Log Workflow Execution

Pattern 3: Build Incremental Data Pipeline

Pattern 4: Bulk CSV Import to Database

Security Model

Project Isolation

Automatic Authentication

Network Security

Best Practices

Performance Considerations

Library Operations

Database Operations

Execution Timeout

Limitations

Limitations

Troubleshooting

Common Errors

Debugging Tips

Next Steps