从Mongo数据库迁移数据到另一个数据库。

huangapple
go评论124阅读模式
英文:

Migrating data from mongo database to another

问题

我有两个数据库连接docDb和mongoDb，每个数据库连接都有三个数据库db1，db2和db3。我想使用mongo-driver在Go中从docDb的db1读取所有数据，并将其写入mongoDb的db1，我也想对其他数据库做同样的操作。

这是我目前的代码：

  1. func MigrateData(mongoConnection *mongo.Client, docDbConnection *mongo.Client){
  2.     db1Doc := docDbConnection.Database("db1")
  3.     db1Mongo := mongoConnection.Database("db1")
  5.     coll_doc := db1Doc.Collection("collection_1")
  6.     cursor, err := coll_doc.Find(context.TODO(), bson.M{})
  7.     if err != nil {
  8.         log.Fatal(err)
  9.     }
  10.     var documents []bson.M
  11.     if err = cursor.All(context.TODO(), &documents); err != nil {
  12.         log.Fatal(err)
  13.     }
  15.     coll_mongo := db1Mongo.Collection("collection_1")
  16. }

现在documents包含了db1中collection_1的所有文档。但是我不知道如何将它们写入mongodb的db1中的collection_1，如果文档已经存在则跳过。我在网上做了一些研究，但发现只有插入自定义文档对象的方法。但是这里我得到的是类型[]primitive.M。

  1. 我该如何将它们写入coll_mongo（mongodb中db1的collection_1）？

  2. 如果我按照这种方式操作，我需要预先知道集合的名称，并在集合之间进行迁移，或者获取集合名称列表并循环遍历，但这听起来像是一种变通方法或错误的方法。在Go和mongo-driver中，是否有另一种方法可以从一个数据库中读取所有内容，并直接将其写入另一个数据库（所有集合，所有文档）？

英文:

I have two database connections docDb and mongoDb each having three databases db1,db2, and db3. I want to read all data from db1 of docDb to db1 of mongoDb in Go using mongo-driver and I want to do the same for other databases as well.

This is what I have so far,

  1. func MigrateData(mongoConnection *mongo.Client, docDbConnection *mongo.Client){
  2. 	db1Doc := docDbConnection.Database("db1")
  3. 	db1Mongo := mongoConnection.Database("db1")
  5. 	coll_doc := db1Doc.Collection("collection_1")
  6. 	cursor, err := coll_doc.Find(context.TODO(), bson.M{})
  7. 	if err != nil {
  8. 		log.Fatal(err)
  9. 	}
  10. 	var documents []bson.M
  11. 	if err = cursor.All(context.TODO(), &documents); err != nil {
  12. 		log.Fatal(err)
  13. 	}
  15.     coll_mongo := db1Mongo.Collection("collection_1")
  16. }

Now this documents consists of all the documents in collection_1 in db1. But I am not able to figure out how to write these to collection_1 of db1 and skip if the document is already present. I did some research online but all I found was inserting custom-made document objects. But here I am getting type []primitive.M.

  1. How do I write these to coll_mongo ( collection_1 in db1 of mongodb)?

  2. If I do it this way, I will need to know the collection names beforehand and migrate in between collections, or get a list of collection names and loop through it but this sounds like a workaround or a wrong way to do it. Is there another way in Go and mongo-driver where I can read everything from a database and directly write it to another database ( all collections, all documents )?

答案1

得分: 1

使用Go执行bash脚本
首先使用以下bash命令将数据库docDb转储：

  1. mongodump --db=docDb --out=./

然后使用以下命令将数据导入到mongoDb中：
创建restore.sh文件并将以下命令放入其中：

  1. mongorestore -d mongoDb -c db1  ./docDb/db1.bson --drop
  2. mongorestore -d mongoDb -c db2  ./docDb/db2.bson --drop 
  3. mongorestore -d mongoDb -c db3  ./docDb/db3.bson --drop

以下是在Go中执行bash命令的示例代码：

  1. package main
  3. import (
  4.    "fmt"
  5.    "os/exec"
  6.    "os"
  7.    "bytes"
  8.    "io"
  9. )
  11. func main() {
  12.     app := "bash backup.sh"
  13.     cmd, err := exec.Run(app, []string{app, "-l"}, nil, "", exec.DevNull, exec.Pipe, exec.Pipe)
  15.     if (err != nil) {
  16.        fmt.Fprintln(os.Stderr, err.String())
  17.        return
  18.     }
  20.     var b bytes.Buffer
  21.     io.Copy(&b, cmd.Stdout)
  22.     fmt.Println(b.String())
  24.     cmd.Close()
  26.     app := "bash restore.sh"
  27.     cmd, err := exec.Run(app, []string{app, "-l"}, nil, "", exec.DevNull, exec.Pipe, exec.Pipe)
  29.     if (err != nil) {
  30.        fmt.Fprintln(os.Stderr, err.String())
  31.        return
  32.     }
  34.     var b bytes.Buffer
  35.     io.Copy(&b, cmd.Stdout)
  36.     fmt.Println(b.String())
  38.     cmd.Close()
  39. }

请注意，以上代码中的backup.sh和restore.sh文件应该与Go代码文件在同一目录下。

英文:

execute bash script with Go
first dump db docDb with this bash command
create backup.sh file and put this command in it

  1. mongodump --db=docDb --out=./

and then import data to mongoDb with these commands
create restore.sh file and put in it

  1. mongorestore -d mongoDb -c db1  ./docDb/db1.bson --drop
  2. mongorestore -d mongoDb -c db2  ./docDb/db2.bson --drop 
  3. mongorestore -d mongoDb -c db3  ./docDb/db3.bson --drop

this is sample code in Go to exec bash command

  1. package main
  3. import (
  4.    "fmt"
  5.    "exec"
  6.    "os"
  7.    "bytes"
  8.    "io"
  9. )
  11. func main() {
  12.     app := "bash backup.sh"
  13.     cmd, err := exec.Run(app, []string{app, "-l"}, nil, "", exec.DevNull, exec.Pipe, exec.Pipe)
  15.     if (err != nil) {
  16.        fmt.Fprintln(os.Stderr, err.String())
  17.        return
  18.     }
  20.     var b bytes.Buffer
  21.     io.Copy(&b, cmd.Stdout)
  22.     fmt.Println(b.String())
  24.     cmd.Close()
  26.     app := "bash restore.sh"
  27.     cmd, err := exec.Run(app, []string{app, "-l"}, nil, "", exec.DevNull, exec.Pipe, exec.Pipe)
  29.     if (err != nil) {
  30.        fmt.Fprintln(os.Stderr, err.String())
  31.        return
  32.     }
  34.     var b bytes.Buffer
  35.     io.Copy(&b, cmd.Stdout)
  36.     fmt.Println(b.String())
  38.     cmd.Close()
  41. }

huangapple
  • 本文由 发表于 2021年8月24日 17:10:34
  • 转载请务必保留本文链接：https://go.coder-hub.com/68904782.html
  • data-migration
  • go
  • mongo-go-driver
  • mongodb
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定