将数据导入到PostgreSQL并导出为CSV文件。

huangapple go评论119阅读模式
英文:

import data into PostgreSQL and export to CSV

问题

我正在尝试使用PostgreSQL进行CSV数据的导入和导出(其中数据库位于远程主机)。通常情况下,我会使用psql命令执行\copy <table> from <local path> ...\copy <table> to <local path> ...,但我需要能够通过Go来执行,而我没有访问shell或没有安装psql的系统。

数据本身预计非常轻量级(可能总共不到2MB的数据),因此我希望库/代码能够推断表的模式并将数据推送到表中。

有关如何实现此功能的任何建议吗?我不确定Go的database/sqlpgxpq是否允许在不指定列的情况下执行此操作。对此有何建议吗?

编辑:

我最终使用了https://github.com/joho/sqltocsv进行数据库导出,这非常简单,我不需要定义任何模式/结构。

我没有代码,但我尝试了gorm并意识到我需要定义一些结构/模式。

英文:

I am trying to do csv import and export data with postgresql (where the data base is at a remote host). Normally I would use the psql command to do \copy &lt;table&gt; from &lt;local path&gt; ... and \copy &lt;table&gt; to &lt;local path&gt; ... but I need to be able to do it via Go where I don't have access to shell or systems that don't have psql installed.

The data itself is expected to be pretty light (maybe < 2 MB of data together), hence I am trying not to implement any structs/schema of track the columns in the tables. When importing into DB, I want to library/code to infer the schema of the table and push the data to the tables.

Any suggestions on how to implement this? I am not sure if any of the Go database/sql or pgx or pq allow this without being able to specify columns. Any advice on this this?

Edit:

I ended up using https://github.com/joho/sqltocsv for DB export, which is pretty simple enough where I don't have to define any schema/structs.

I don't have the code but I tried gorm and realized I need to define some struct/schema for it.

答案1

得分: 3

我找到了使用pgx包的方法来实现(感谢@Gustavo Kawamoto的建议)。
这是我的导入和导出代码:

  1. package main
  2. import (
  3. "fmt"
  4. "os"
  5. "github.com/jackc/pgx"
  6. )
  7. func main() {
  8. pgxConConfig := pgx.ConnConfig{
  9. Port: 5432,
  10. Host: "remote_host",
  11. Database: "db_name",
  12. User: "my_user",
  13. Password: "my_password",
  14. }
  15. conn, err := pgx.Connect(pgxConConfig)
  16. if err != nil {
  17. panic(err)
  18. }
  19. defer conn.Close()
  20. tables := []string{"table1", "table2", "table3"}
  21. import_dir := "/dir_to_import_from"
  22. export_dir := "/dir_to_export_to"
  23. for _, t := range tables {
  24. f, err := os.OpenFile(fmt.Sprintf("%s/table_%s.csv", import_dir, t), os.O_RDONLY, 0777)
  25. if err != nil {
  26. return
  27. }
  28. f.Close()
  29. err = importer(conn, f, t)
  30. if err != nil {
  31. break
  32. }
  33. fmt.Println(" Done with import and doing export")
  34. ef, err := os.OpenFile(fmt.Sprintf("%s/table_%s.csv", export_dir, t), os.O_CREATE|os.O_WRONLY, 0777)
  35. if err != nil {
  36. fmt.Println("error opening file:", err)
  37. return
  38. }
  39. ef.Close()
  40. err = exporter(conn, ef, t)
  41. if err != nil {
  42. break
  43. }
  44. }
  45. }
  46. func importer(conn *pgx.Conn, f *os.File, table string) error {
  47. res, err := conn.CopyFromReader(f, fmt.Sprintf("COPY %s FROM STDIN DELIMITER '|' CSV HEADER", table))
  48. if err != nil {
  49. return err
  50. }
  51. fmt.Println("==> import rows affected:", res.RowsAffected())
  52. return nil
  53. }
  54. func exporter(conn *pgx.Conn, f *os.File, table string) error {
  55. res, err := conn.CopyToWriter(f, fmt.Sprintf("COPY %s TO STDOUT DELIMITER '|' CSV HEADER", table))
  56. if err != nil {
  57. return fmt.Errorf("error exporting file: %+v", err)
  58. }
  59. fmt.Println("==> export rows affected:", res.RowsAffected())
  60. return nil
  61. }
英文:

I found way to do it with pgx package (thanks to @Gustavo Kawamoto suggestion).
Here's my import and export:

  1. package main
  2. import (
  3. &quot;fmt&quot;
  4. &quot;os&quot;
  5. &quot;github.com/jackc/pgx&quot;
  6. )
  7. func main() {
  8. pgxConConfig := pgx.ConnConfig{
  9. Port: 5432,
  10. Host: &quot;remote_host&quot;,
  11. Database: &quot;db_name&quot;,
  12. User: &quot;my_user&quot;,
  13. Password: &quot;my_password&quot;,
  14. }
  15. conn, err := pgx.Connect(pgxConConfig)
  16. if err != nil {
  17. panic(err)
  18. }
  19. defer conn.Close()
  20. tables := []string{&quot;table1&quot;, &quot;table2&quot;, &quot;table3&quot;,}
  21. import_dir := &quot;/dir_to_import_from&quot;
  22. export_dir := &quot;/dir_to_export_to&quot;
  23. for _, t := range tables {
  24. f, err := os.OpenFile(fmt.Sprintf(&quot;%s/table_%s.csv&quot;, import_dir, t), os.O_RDONLY, 0777)
  25. if err != nil {
  26. return
  27. }
  28. f.Close()
  29. err = importer(conn, f, t)
  30. if err != nil {
  31. break
  32. }
  33. fmt.Println(&quot; Done with import and doing export&quot;)
  34. ef, err := os.OpenFile(fmt.Sprintf(&quot;%s/table_%s.csv&quot;, export_dir, t), os.O_CREATE|os.O_WRONLY, 0777)
  35. if err != nil {
  36. fmt.Println(&quot;error opening file:&quot;, err)
  37. return
  38. }
  39. ef.Close()
  40. err = exporter(conn, ef, t)
  41. if err != nil {
  42. break
  43. }
  44. }
  45. }
  46. func importer(conn *pgx.Conn, f *os.File, table string) error {
  47. res, err := conn.CopyFromReader(f, fmt.Sprintf(&quot;COPY %s FROM STDIN DELIMITER &#39;|&#39; CSV HEADER&quot;, table))
  48. if err != nil {
  49. return err
  50. }
  51. fmt.Println(&quot;==&gt; import rows affected:&quot;, res.RowsAffected())
  52. return nil
  53. }
  54. func exporter(conn *pgx.Conn, f *os.File, table string) error {
  55. res, err := conn.CopyToWriter(f, fmt.Sprintf(&quot;COPY %s TO STDOUT DELIMITER &#39;|&#39; CSV HEADER&quot;, table))
  56. if err != nil {
  57. return fmt.Errorf(&quot;error exporting file: %+v&quot;, err)
  58. }
  59. fmt.Println(&quot;==&gt; export rows affected:&quot;, res.RowsAffected())
  60. return nil
  61. }

答案2

得分: 2

请查看此软件包:https://github.com/chop-dbhi/sql-importer

  1. 自动创建表格
  2. 唯一性和非空检测。
  3. 支持超过1600列的CSV文件(Postgres的限制)
英文:

Checkout this pkg: https://github.com/chop-dbhi/sql-importer

  1. Automatic table creation
  2. Uniqueness and not null detection.
  3. Support for CSV files wider than 1600 columns (the Postgres limit)

huangapple
  • 本文由 发表于 2021年8月9日 14:30:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/68707706.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定