在使用fs.createWriteStream将数据写入BigQuery时出现模式错误(Node.js)。

huangapple go评论124阅读模式
英文:

schema error when using fs.createWriteStream to write data to bigquery (node.js)

问题

错误:作业或表上未指定模式

不知道为什么会发生这个错误。代码来自文档。我还尝试了以下不同格式的方式,如 fs.createWriteStream({sourceFormat: "json"}),但结果仍然是相同的错误。

const { BigQuery } = require("@google-cloud/bigquery");
const bigquery = new BigQuery();
const dataset = bigquery.dataset("firebase_test_data");
const table = dataset.table("flattened_data");
const fs = require("fs");

fs.createReadStream("./data.json")
  .pipe(table.createWriteStream("json"))
  .on("job", (job) => {
    // `job` 是一个可用于检查请求状态的 Job 对象。
    console.log(job);
  })
  .on("complete", (job) => {
    // 作业已成功完成。
  });
英文:

Error : No schema specified on job or table.

No idea why this error is happening. The code is from documentation. I have also tried following a different format such fs.createWriteStream({sourceFormat: "json"}) - but it results in the same error.

const { BigQuery } = require("@google-cloud/bigquery");
const bigquery = new BigQuery();
const dataset = bigquery.dataset("firebase_test_data");
const table = dataset.table("flattened_data");
const fs = require("fs");

fs.createReadStream("./data.json")
  .pipe(table.createWriteStream("json"))
  .on("job", (job) => {
    // `job` is a Job object that can be used to check the status of the
    // request.
    console.log(job);
  })
  .on("complete", (job) => {
    // The job has completed successfully.
  });

答案1

得分: 0

const table = dataset.table("flattened_data"); 中定义的表格由于没有正确的模式而导致出现此错误,而您正在传递给data.json的模式是不合适的。

根据Google文档尝试了以下代码,并在BigQuery中指定了表格的模式,成功将数据加载到表格中。

const { BigQuery } = require('@google-cloud/bigquery');
const bigquery = new BigQuery();
const dataset = bigquery.dataset('my-dataset');
const table = dataset.table('my-table');

//-
// 从JSON文件加载数据。
//-
const fs = require('fs');

fs.createReadStream('/path-to-json/data.json')
 .pipe(table.createWriteStream('json'))
 .on('job', (job) => {
   // `job`是一个Job对象,可用于检查请求的状态。
 })
 .on('complete', (job) => {
   // 作业已成功完成。
 });

<details>
<summary>英文:</summary>

was getting this error since the table defined in `const table = dataset.table(&quot;flattened_data&quot;);`  doesn’t have the appropriate schema which you are passing in data.json.

tried the following code as per the [Google documentation][1] and specified the schema of table in BigQuery which successfully loads data into the table.


    const {BigQuery} = require(&#39;@google-cloud/bigquery&#39;);
    const bigquery = new BigQuery();
    const dataset = bigquery.dataset(&#39;my-dataset&#39;);
    const table = dataset.table(&#39;my-table&#39;);
    
    //-
    // Load data from a JSON file.
    //-
    const fs = require(&#39;fs&#39;);
    
    fs.createReadStream(&#39;/path-to-json/data.json&#39;)
     .pipe(table.createWriteStream(&#39;json&#39;))
     .on(&#39;job&#39;, (job) =&gt; {
       // `job` is a Job object that can be used to check the status of the
       // request.
     })
     .on(&#39;complete&#39;, (job) =&gt; {
       // The job has completed successfully.
     });


  [1]: https://googleapis.dev/nodejs/bigquery/latest/Table.html#createWriteStream-examples

</details>



huangapple
  • 本文由 发表于 2023年1月7日 02:03:50
  • 转载请务必保留本文链接:https://go.coder-hub.com/75034557.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定