Skip to main content

Step 5: Create Data Mirror Task

Creating Your First Data Mirror Task​

Now that you have set up your source database and ClickHouse target, you're ready to create a data mirror task. This task will continuously synchronize data from your source to ClickHouse, keeping your analytics data up-to-date in real-time.

Step-by-Step Guide​

1. Navigate to the Tasks Section​

In the ClickPipes dashboard, click on Tasks in the left navigation menu, then click the Create button.

2. Select Source and Target​

In the task creation wizard:

  1. Select your source connection
  2. Select your ClickHouse target connection

select_source_target

3. Select Tables to Mirror​

Choose which tables you want to mirror from your source database:

  1. You can select individual tables or use the Select All option
  2. For each table, you can customize:
    • Which columns to include
    • Whether to sync full data or incremental changes only
    • Transformation rules (optional)

select_tables

4. Configure Mirror Settings​

Configure various task-related settings, such as:

  1. Performance Configuration:
    • Source Table Concurrency: Control the number of tables processed simultaneously
    • Single Table Read Concurrency: Optimize reading performance for large tables
    • Target Write Batch Size: Adjust batch sizes to balance performance and resource usage
  2. Resources and Monitoring:
    • Memory Usage: Set memory limits for the task
    • Scheduled Execution: Configure task execution time and frequency
    • Task Alerts: Set up notification rules for exception scenarios

config_mirror

5. Start the Task​

Review your settings and click Start Task to begin the synchronization process.

6. Monitor Progress​

Once your task is running, you can monitor its progress:

  1. The dashboard shows real-time metrics including:
    • Records processed
    • Current throughput
    • Sync status for each table
    • Latency

mirror_monitor

  1. Wait for the initial synchronization to complete. Depending on your data volume, this could take from minutes to hours.

Next Steps​

After your data is successfully mirrored to ClickHouse, you can:

  1. Query your data directly in ClickHouse for analytics
  2. Create Data APIs to expose your data through HTTP endpoints
  3. Connect BI tools to your ClickHouse instance for visualization