MongoDB

MongoDB is a popular open-source NoSQL database that stores and retrieves data in a flexible and scalable manner

This article provides a comprehensive guide on how to add MongoDB to ClickPipes, enabling you to leverage its scalability, flexibility, querying, and indexing capabilities for your data processing needs.

Supported Versions and Architectures

Category	Description
Version	MongoDB 3.6 and above. For MongoDB 3.4 and below, please choose the data source named MongoDB Below 3.4.
Architecture	Supports replica set and sharded cluster architectures. Additionally, full and incremental data synchronization can be performed from the secondary nodes of a sharded cluster

Supported Data Types

Category	Data Types
Strings and Code	String, JavaScript, Symbol
Numeric Types	Double, Int32, Int64, Decimal128
Document and Array	Document, Array
Binary and ObjectId	Binary, ObjectId
Boolean Type	Boolean
Date and Timestamp	Date, Timestamp
Special Types	Min Key, Max Key, Null

Prerequisites

1. Make sure that the schema of the source database is a replica set or a sharding cluster. If it is standalone, you can configure it as a single-member replica set to open Oplog. For more information, see Convert a Standalone to a Replica Set.

2. To ensure sufficient storage space for the Oplog, it is important to configure it to accommodate at least 24 hours' worth of data. For detailed instructions, see Change the Size of the Oplog.

3. To create an account and grant permissions according to permission management requirements, follow the necessary steps.

   Grant Read Access to Specific Databases

   use admin
   db.createUser(
     {
       user: "ClickPipes",
       pwd: "my_password",
       roles: [
          { role: "read", db: "database_name" },
          { role: "read", db: "local" },
          { role: "read", db: "config" },
          { role: "clusterMonitor", db: "admin" },
       ]
     }
   )

   Grant Read Access to All Databases

   use admin
   db.createUser(
     {
       user: "clickpipes",
       pwd: "my_password",
       roles: [
          { role: "readAnyDatabase", db: "admin" },
          { role: "clusterMonitor", db: "admin" },
       ]
     }
   )
   tip

   In shard cluster architectures, the shard server is unable to retrieve user permissions from the config database. Therefore, it is necessary to create corresponding users and grant permissions on the master nodes of each shard.

4. When the source database is a cluster, in order to improve data synchronization performance, ClickPipes Cloud will create a thread for each shard and read the data. Before configuring data synchronization/development tasks, you also need to perform the following operations.

5. For sharded cluster architectures, to improve data synchronization performance, ClickPipes will create a thread for each shard to read data. Before configuring data synchronization/development tasks, you also need to perform the following operations:

   1. Stop the Balancer to avoid data inconsistency caused by chunk migrations.
   2. Clean up orphaned documents to avoid _id conflicts.

Source Config

• Connection Settings
  • Name: Fill in a unique name that has business significance.
  • Type: Supports MongoDB as a source or target database.
  • Connection Mode: Choose how you want to connect:
    • URI Mode: After selecting this mode, you will be required to provide the necessary information for the database URI connection. The connection string should include the username and password, which are concatenated in the format. For example, the connection string may look like: mongodb://admin:password@192.168.0.100:27017/mydb?replicaSet=xxx&authSource=admin.
    • Standard mode: After selecting this mode, you need to fill in the database address, name, account number, password and other connection string parameters.