PostgreSQL 使用 Python 加密时的 UPSERT。

huangapple go评论65阅读模式
英文:

Postgresql UPSERT while encrypting column by python encryption

问题

I want to call my encryption_fnc in DO UPDATE SET section, while it updates, I also want to do encryption.

我想在“DO UPDATE SET”部分调用我的encryption_fnc,在更新时也要进行加密。

英文:

I am using BULK UPSERT to insert record into new table while data of column is being encrypting by python lib CryptoDome.
here is what I done

def bulk_upsert_query(data, table_name=None):
    values_list = []
    add = values_list.append

    for i in data:
        encrypt_user_id = encryption_fnc(str(i.user_id))

        add(f"({i.id},'{encrypt_user_id}','{i.name}')")

    upsert_sql = r"""insert into upsert_test_table 
                    (id,encrypt_user_id,name)
                   values
                   """ + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT 
                  "upsert_test_table_pkey"
                   DO UPDATE SET
                   name=excluded.name,
                   encrypt_user_id=excluded.user_id"""
    return upsert_sql

data = data from my old table that have user_id as int(ex: 124,345,786)
in upsert_test_table table id column is primary key column.
this bulk_upsert_query fnc will encrypt int user_id and append to values_list then create a upsert query.

As per insertion it works as I expected, but when it comes for update if conflict with id column then as you can see I set encrypt_user_id=excluded.user_id for user_id column,
it update existing encrypted user_id with int user_id(from old table), because I did not put any encryption fnc here,
Update: user_id column is changeable

So what I want is I want to call my encryption_fnc in DO UPDATE SET section,
while it update I also want to do encryption.

Can any one tell me who can I achieve this?

Note: I can do encryption in database site by using pgcrypto but my requirement is do encryption in python side not database side.

答案1

得分: 1

以下是翻译好的部分:

根据评论
在插入负载中没有user_id列,因此excluded.user_idnull

ON CONFLICT...DO UPDATE SET encrypt_user_id=excluded.user_id 插入的是null而不是旧的、未加密的user_id演示

CREATE TABLE upsert_test_table 
  (id SMALLSERIAL,
  encrypt_user_id TEXT,
  name TEXT,
  user_id INT,
  CONSTRAINT "upsert_test_table_pkey" PRIMARY KEY (id));
INSERT INTO upsert_test_table VALUES 
  (1,'enc1','Bob',11);

INSERT INTO upsert_test_table (id,encrypt_user_id,name) VALUES
  (1,'enc101','Bob2'),--这里有冲突
  (2,'enc2','Ted')
ON CONFLICT ON CONSTRAINT "upsert_test_table_pkey"
DO UPDATE SET 
  name = excluded.name,              --名字将被传入的数据覆盖
  encrypt_user_id = excluded.user_id;--null,因为user_id不在传入的数据中

SELECT * FROM upsert_test_table;
-- id | encrypt_user_id | name | user_id
------+-----------------+------+---------
--  1 |                 | Bob2 |      11
--  2 | enc2            | Ted  |

文档中:

> ON CONFLICT DO UPDATE 中的 SETWHERE 子句可以使用表的名称(或别名)访问现有行,并使用特殊的 excluded 表访问插入建议的行。

问题在于excluded被误解,而被误用为冲突时表中已存在的数据,实际上它是即将插入的数据,作为该插入的有效负载。

只需纠正这个误解,并将新的/即将插入的值称为 excluded,将旧的/已存在的称为目标表的名称和列即可。

upsert_sql = r"""insert into upsert_test_table 
                (id,encrypt_user_id,name)
               values
               """ + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT 
              "upsert_test_table_pkey"
               DO UPDATE SET
               name = upsert_test_table.name,             --保留旧的名字
               encrypt_user_id = excluded.encrypt_user_id --用新的覆盖
              """
英文:

As per the comment:
there is no user_id column in that insert payload, so excluded.user_id is null.

ON CONFLICT...DO UPDATE SET encrypt_user_id=excluded.user_id is inserting a null rather than the old, unencrypted user_id: demo

CREATE TABLE upsert_test_table 
  (id SMALLSERIAL,
  encrypt_user_id TEXT,
  name TEXT,
  user_id INT,
  CONSTRAINT "upsert_test_table_pkey" PRIMARY KEY (id));
INSERT INTO upsert_test_table VALUES 
  (1,'enc1','Bob',11);

INSERT INTO upsert_test_table (id,encrypt_user_id,name) VALUES
  (1,'enc101','Bob2'),--conflict here
  (2,'enc2','Ted')
ON CONFLICT ON CONSTRAINT "upsert_test_table_pkey"
DO UPDATE SET 
  name = excluded.name,              --name will be overwritten by incoming data
  encrypt_user_id = excluded.user_id;--null because user_id not in incoming data

SELECT * FROM upsert_test_table;
-- id | encrypt_user_id | name | user_id
------+-----------------+------+---------
--  1 |                 | Bob2 |      11
--  2 | enc2            | Ted  |

From the documentation:

> The SET and WHERE clauses in ON CONFLICT DO UPDATE have access to the existing row using the table's name (or an alias), and to the row proposed for insertion using the special excluded table.

The issue here was that excluded was misunderstood and instead used as the preexisting data found already in the table upon conflict, while it's actually the data that was coming in, as the payload of that insert.

It should be enough to correct that one misinterpretation and refer to new/incoming/inserted value as excluded, old/existing by the target table name and column:

upsert_sql = r"""insert into upsert_test_table 
                (id,encrypt_user_id,name)
               values
               """ + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT 
              "upsert_test_table_pkey"
               DO UPDATE SET
               name = upsert_test_table.name,             --keep the old name
               encrypt_user_id = excluded.encrypt_user_id --overwrite with new
              """

答案2

得分: -1

I apologize for any misunderstanding earlier. Here is the translated portion of the text:

抱歉,我刚才理解错了,我以为excluded表是原始(输入)表,但当我用excluded.user_id替换为excluded.encrypt_user_id时,它起作用了,尽管我不知道excluded表是如何工作的。

upsert_sql = r""""insert into upsert_test_table 
                    (id,encrypt_user_id,name)
                   values
                   """ + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT 
                  "upsert_test_table_pkey"
                   DO UPDATE SET
                   name=excluded.name,
                   encrypt_user_id=excluded.encrypt_user_id""""
英文:

Solved:
Sorry I was stupid, I thought excluded table is original(input payload) table, but when I replaced excluded.user_id with excluded.encrypt_user_id and it works, although have no idea how excluded table works.

upsert_sql = r"""insert into upsert_test_table 
                    (id,encrypt_user_id,name)
                   values
                   """ + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT 
                  "upsert_test_table_pkey"
                   DO UPDATE SET
                   name=excluded.name,
                   encrypt_user_id=excluded.encrypt_user_id"""

huangapple
  • 本文由 发表于 2023年4月19日 16:01:53
  • 转载请务必保留本文链接:https://go.coder-hub.com/76052051.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定