从Mongo数据库迁移数据到另一个数据库。

huangapple go评论78阅读模式
英文:

Migrating data from mongo database to another

问题

我有两个数据库连接docDbmongoDb,每个数据库连接都有三个数据库db1db2db3。我想使用mongo-driver在Go中从docDbdb1读取所有数据,并将其写入mongoDbdb1,我也想对其他数据库做同样的操作。

这是我目前的代码:

func MigrateData(mongoConnection *mongo.Client, docDbConnection *mongo.Client){
    db1Doc := docDbConnection.Database("db1")
    db1Mongo := mongoConnection.Database("db1")

    coll_doc := db1Doc.Collection("collection_1")
    cursor, err := coll_doc.Find(context.TODO(), bson.M{})
    if err != nil {
        log.Fatal(err)
    }
    var documents []bson.M
    if err = cursor.All(context.TODO(), &documents); err != nil {
        log.Fatal(err)
    }

    coll_mongo := db1Mongo.Collection("collection_1")
}

现在documents包含了db1collection_1的所有文档。但是我不知道如何将它们写入mongodbdb1中的collection_1,如果文档已经存在则跳过。我在网上做了一些研究,但发现只有插入自定义文档对象的方法。但是这里我得到的是类型[]primitive.M

  1. 我该如何将它们写入coll_mongomongodbdb1collection_1)?

  2. 如果我按照这种方式操作,我需要预先知道集合的名称,并在集合之间进行迁移,或者获取集合名称列表并循环遍历,但这听起来像是一种变通方法或错误的方法。在Go和mongo-driver中,是否有另一种方法可以从一个数据库中读取所有内容,并直接将其写入另一个数据库(所有集合,所有文档)?

英文:

I have two database connections docDb and mongoDb each having three databases db1,db2, and db3. I want to read all data from db1 of docDb to db1 of mongoDb in Go using mongo-driver and I want to do the same for other databases as well.

This is what I have so far,

func MigrateData(mongoConnection *mongo.Client, docDbConnection *mongo.Client){
	db1Doc := docDbConnection.Database("db1")
	db1Mongo := mongoConnection.Database("db1")

	coll_doc := db1Doc.Collection("collection_1")
	cursor, err := coll_doc.Find(context.TODO(), bson.M{})
	if err != nil {
		log.Fatal(err)
	}
	var documents []bson.M
	if err = cursor.All(context.TODO(), &documents); err != nil {
		log.Fatal(err)
	}

    coll_mongo := db1Mongo.Collection("collection_1")
}

Now this documents consists of all the documents in collection_1 in db1. But I am not able to figure out how to write these to collection_1 of db1 and skip if the document is already present. I did some research online but all I found was inserting custom-made document objects. But here I am getting type []primitive.M.

  1. How do I write these to coll_mongo ( collection_1 in db1 of mongodb)?

  2. If I do it this way, I will need to know the collection names beforehand and migrate in between collections, or get a list of collection names and loop through it but this sounds like a workaround or a wrong way to do it. Is there another way in Go and mongo-driver where I can read everything from a database and directly write it to another database ( all collections, all documents )?

答案1

得分: 1

使用Go执行bash脚本
首先使用以下bash命令将数据库docDb转储:

mongodump --db=docDb --out=./

然后使用以下命令将数据导入到mongoDb中:
创建restore.sh文件并将以下命令放入其中:

mongorestore -d mongoDb -c db1  ./docDb/db1.bson --drop
mongorestore -d mongoDb -c db2  ./docDb/db2.bson --drop 
mongorestore -d mongoDb -c db3  ./docDb/db3.bson --drop 

以下是在Go中执行bash命令的示例代码:

package main

import (
   "fmt"
   "os/exec"
   "os"
   "bytes"
   "io"
)

func main() {
    app := "bash backup.sh"
    cmd, err := exec.Run(app, []string{app, "-l"}, nil, "", exec.DevNull, exec.Pipe, exec.Pipe)

    if (err != nil) {
       fmt.Fprintln(os.Stderr, err.String())
       return
    }

    var b bytes.Buffer
    io.Copy(&b, cmd.Stdout)
    fmt.Println(b.String())

    cmd.Close()

    app := "bash restore.sh"
    cmd, err := exec.Run(app, []string{app, "-l"}, nil, "", exec.DevNull, exec.Pipe, exec.Pipe)

    if (err != nil) {
       fmt.Fprintln(os.Stderr, err.String())
       return
    }

    var b bytes.Buffer
    io.Copy(&b, cmd.Stdout)
    fmt.Println(b.String())

    cmd.Close()
}

请注意,以上代码中的backup.shrestore.sh文件应该与Go代码文件在同一目录下。

英文:

execute bash script with Go
first dump db docDb with this bash command
create backup.sh file and put this command in it

mongodump --db=docDb --out=./

and then import data to mongoDb with these commands
create restore.sh file and put in it

mongorestore -d mongoDb -c db1  ./docDb/db1.bson --drop
mongorestore -d mongoDb -c db2  ./docDb/db2.bson --drop 
mongorestore -d mongoDb -c db3  ./docDb/db3.bson --drop 

this is sample code in Go to exec bash command

package main

import (
   "fmt"
   "exec"
   "os"
   "bytes"
   "io"
)

func main() {
    app := "bash backup.sh"
    cmd, err := exec.Run(app, []string{app, "-l"}, nil, "", exec.DevNull, exec.Pipe, exec.Pipe)

    if (err != nil) {
       fmt.Fprintln(os.Stderr, err.String())
       return
    }

    var b bytes.Buffer
    io.Copy(&b, cmd.Stdout)
    fmt.Println(b.String())

    cmd.Close()

    app := "bash restore.sh"
    cmd, err := exec.Run(app, []string{app, "-l"}, nil, "", exec.DevNull, exec.Pipe, exec.Pipe)

    if (err != nil) {
       fmt.Fprintln(os.Stderr, err.String())
       return
    }

    var b bytes.Buffer
    io.Copy(&b, cmd.Stdout)
    fmt.Println(b.String())

    cmd.Close()


}

huangapple
  • 本文由 发表于 2021年8月24日 17:10:34
  • 转载请务必保留本文链接:https://go.coder-hub.com/68904782.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定