在尝试使用Node.js从MongoDB导入数据到Elasticsearch时遇到错误。

huangapple go评论73阅读模式
英文:

Getting error while trying to Import data from MongoDB to ElasticSearch using Nodejs

问题

我正在尝试从MongoDB导入小型文档到ElasticSearch,但出现错误:

{
  "index": {
    "_index": "impact-fulltext",
    "_id": "t2oLkoUBwNXsTufYzszL",
    "status": 400,
    "error": {
      "type": "mapper_parsing_exception",
      "reason": "failed to parse field [_id] of type [_id] in document with id 't2oLkoUBwNXsTufYzszL'. Preview of field's value: '605315a3b4f719d00f69f2d3'",
      "caused_by": {
        "type": "mapper_parsing_exception",
        "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
      }
    }
  }
}

我感到困惑,因为我已经定义了_id,但仍然出现错误。

db.collection("article_beta")
  .find()
  .limit(100)
  .toArray((err, docs) => {
    if (err) throw err;

    esClient.bulk(
      {
        body: docs.flatMap((doc) => [
          {
            index: {
              _index: "impact-fulltext",
              _id: doc._id.$oid,
            },
          },
          doc,
        ]),
      },
      (err, resp) => {
        if (err) throw err;
        console.log(resp);
        client.close();
      }
    );
  });
英文:

I am trying to import small documents from MongoDB to ElasticSearch but getting an error

      {
    "index": {
      "_index": "impact-fulltext",
      "_id": "t2oLkoUBwNXsTufYzszL",
      "status": 400,
      "error": {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [_id] of type [_id] in document with id \u0027t2oLkoUBwNXsTufYzszL\u0027. Preview of field\u0027s value: \u0027605315a3b4f719d00f69f2d3\u0027",
        "caused_by": {
          "type": "mapper_parsing_exception",
          "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
        }
      }
    }
  }

I am clueless as I also defined the _id but still getting the error.

    db.collection("article_beta")
  .find()
  .limit(100)
  .toArray((err, docs) => {
    if (err) throw err;

    esClient.bulk(
      {
        body: docs.flatMap((doc) => [
          {
            index: {
              _index: "impact-fulltext",
              _id: doc._id.$oid,
            },
          },
          doc,
        ]),
      },
      (err, resp) => {
        if (err) throw err;
        console.log(resp);
        client.close();
      }
    );
  });

答案1

得分: 2

在Elasticsearch中,文档包含一些元信息和文档的源数据。因此,文档的结构如下:

{
   "_index": "index-name",
   "_id": "document-id-stored-by-elasticsearch",
   "_source": {
      // 文档本身
   }
}

因此,在文档内部,您不能像下面这样使用_id字段,因为_id字段是保留字段:

POST myindex/_doc
{
  "_id": "123123",
  "field1": "value1"
}

所以这个请求会产生以下错误:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse field [_id] of type [_id] in document with id '16Y6koUBnxYxC21BP-tS'. Preview of field's value: '123123'",
    "caused_by": {
      "type": "mapper_parsing_exception",
      "reason": "Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters."
    }
  },
  "status": 400
}

在您的示例中,doc变量的值来自MongoDB,其中有一个字段名为_id。您需要将该字段的名称更改为id。您可以将以下代码放在flatMap函数内的匿名函数中:

db.collection("article_beta")
  .find()
  .limit(100)
  .toArray((err, docs) => {
    if (err) throw err;

    esClient.bulk(
      {
        body: docs.flatMap((doc) => {
          doc.id = doc._id  // <---- 添加的代码
          delete doc._id    // <---- 添加的代码
          return  [
            {
              index: {
                _index: "impact-fulltext",
                _id: doc.id.$oid,
              },
            },
            doc,
          ]
        }),
      },
      (err, resp) => {
        if (err) throw err;
        console.log(resp);
        client.close();
      }
    );
  });
英文:

In elasticsearch, the documents have some meta information and also the source of the documents. So, the document structure is following :

{
   &quot;_index&quot;: &quot;index-name&quot;,
   &quot;_id&quot;: &quot;document-id-stored-by-elasticsearch&quot;,
   &quot;_source&quot;: {
      // documents itself
   }
}

So, inside the document, you can not use _id field as follow because _id field is reserved field :

POST myindex/_doc
{
  &quot;_id&quot;: &quot;123123&quot;,
  &quot;field1&quot;: &quot;value1&quot;
}

So this request will give an error as follow :

{
  &quot;error&quot;: {
    &quot;root_cause&quot;: [
      {
        &quot;type&quot;: &quot;mapper_parsing_exception&quot;,
        &quot;reason&quot;: &quot;Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters.&quot;
      }
    ],
    &quot;type&quot;: &quot;mapper_parsing_exception&quot;,
    &quot;reason&quot;: &quot;failed to parse field [_id] of type [_id] in document with id &#39;16Y6koUBnxYxC21BP-tS&#39;. Preview of field&#39;s value: &#39;123123&#39;&quot;,
    &quot;caused_by&quot;: {
      &quot;type&quot;: &quot;mapper_parsing_exception&quot;,
      &quot;reason&quot;: &quot;Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters.&quot;
    }
  },
  &quot;status&quot;: 400
}

In your example, doc variable which the value coming from Mongodb have a field as _id. You need to change the name of that field as id. You can put following lines inside of your anonymous function inside the flatMap function.

db.collection(&quot;article_beta&quot;)
  .find()
  .limit(100)
  .toArray((err, docs) =&gt; {
    if (err) throw err;

    esClient.bulk(
      {
        body: docs.flatMap((doc) =&gt; {
          doc.id = doc._id  // &lt;---- added
          delete doc._id    // &lt;---- added
          return  [
            {
              index: {
                _index: &quot;impact-fulltext&quot;,
                _id: doc.id.$oid,
              },
            },
            doc,
          ]
        }),
      },
      (err, resp) =&gt; {
        if (err) throw err;
        console.log(resp);
        client.close();
      }
    );
  });

huangapple
  • 本文由 发表于 2023年1月8日 23:45:49
  • 转载请务必保留本文链接:https://go.coder-hub.com/75049138.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定