英文:
Postgresql UPSERT while encrypting column by python encryption
问题
I want to call my encryption_fnc in DO UPDATE SET section, while it updates, I also want to do encryption.
我想在“DO UPDATE SET”部分调用我的encryption_fnc,在更新时也要进行加密。
英文:
I am using BULK UPSERT to insert record into new table while data of column is being encrypting by python lib CryptoDome.
here is what I done
def bulk_upsert_query(data, table_name=None):
values_list = []
add = values_list.append
for i in data:
encrypt_user_id = encryption_fnc(str(i.user_id))
add(f"({i.id},'{encrypt_user_id}','{i.name}')")
upsert_sql = r"""insert into upsert_test_table
(id,encrypt_user_id,name)
values
""" + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT
"upsert_test_table_pkey"
DO UPDATE SET
name=excluded.name,
encrypt_user_id=excluded.user_id"""
return upsert_sql
data = data from my old table that have user_id as int(ex: 124,345,786)
in upsert_test_table table id column is primary key column.
this bulk_upsert_query fnc will encrypt int user_id and append to values_list then create a upsert query.
As per insertion it works as I expected, but when it comes for update if conflict with id column then as you can see I set encrypt_user_id=excluded.user_id for user_id column,
it update existing encrypted user_id with int user_id(from old table), because I did not put any encryption fnc here,
Update: user_id column is changeable
So what I want is I want to call my encryption_fnc in DO UPDATE SET section,
while it update I also want to do encryption.
Can any one tell me who can I achieve this?
Note: I can do encryption in database site by using pgcrypto but my requirement is do encryption in python side not database side.
答案1
得分: 1
以下是翻译好的部分:
根据评论:
在插入负载中没有user_id列,因此excluded.user_id为null。
ON CONFLICT...DO UPDATE SET encrypt_user_id=excluded.user_id 插入的是null而不是旧的、未加密的user_id:演示
CREATE TABLE upsert_test_table
(id SMALLSERIAL,
encrypt_user_id TEXT,
name TEXT,
user_id INT,
CONSTRAINT "upsert_test_table_pkey" PRIMARY KEY (id));
INSERT INTO upsert_test_table VALUES
(1,'enc1','Bob',11);
INSERT INTO upsert_test_table (id,encrypt_user_id,name) VALUES
(1,'enc101','Bob2'),--这里有冲突
(2,'enc2','Ted')
ON CONFLICT ON CONSTRAINT "upsert_test_table_pkey"
DO UPDATE SET
name = excluded.name, --名字将被传入的数据覆盖
encrypt_user_id = excluded.user_id;--null,因为user_id不在传入的数据中
SELECT * FROM upsert_test_table;
-- id | encrypt_user_id | name | user_id
------+-----------------+------+---------
-- 1 | | Bob2 | 11
-- 2 | enc2 | Ted |
从文档中:
> ON CONFLICT DO UPDATE 中的 SET 和 WHERE 子句可以使用表的名称(或别名)访问现有行,并使用特殊的 excluded 表访问插入建议的行。
问题在于excluded被误解,而被误用为冲突时表中已存在的数据,实际上它是即将插入的数据,作为该插入的有效负载。
只需纠正这个误解,并将新的/即将插入的值称为 excluded,将旧的/已存在的称为目标表的名称和列即可。
upsert_sql = r"""insert into upsert_test_table
(id,encrypt_user_id,name)
values
""" + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT
"upsert_test_table_pkey"
DO UPDATE SET
name = upsert_test_table.name, --保留旧的名字
encrypt_user_id = excluded.encrypt_user_id --用新的覆盖
"""
英文:
As per the comment:
there is no user_id column in that insert payload, so excluded.user_id is null.
ON CONFLICT...DO UPDATE SET encrypt_user_id=excluded.user_id is inserting a null rather than the old, unencrypted user_id: demo
CREATE TABLE upsert_test_table
(id SMALLSERIAL,
encrypt_user_id TEXT,
name TEXT,
user_id INT,
CONSTRAINT "upsert_test_table_pkey" PRIMARY KEY (id));
INSERT INTO upsert_test_table VALUES
(1,'enc1','Bob',11);
INSERT INTO upsert_test_table (id,encrypt_user_id,name) VALUES
(1,'enc101','Bob2'),--conflict here
(2,'enc2','Ted')
ON CONFLICT ON CONSTRAINT "upsert_test_table_pkey"
DO UPDATE SET
name = excluded.name, --name will be overwritten by incoming data
encrypt_user_id = excluded.user_id;--null because user_id not in incoming data
SELECT * FROM upsert_test_table;
-- id | encrypt_user_id | name | user_id
------+-----------------+------+---------
-- 1 | | Bob2 | 11
-- 2 | enc2 | Ted |
From the documentation:
> The SET and WHERE clauses in ON CONFLICT DO UPDATE have access to the existing row using the table's name (or an alias), and to the row proposed for insertion using the special excluded table.
The issue here was that excluded was misunderstood and instead used as the preexisting data found already in the table upon conflict, while it's actually the data that was coming in, as the payload of that insert.
It should be enough to correct that one misinterpretation and refer to new/incoming/inserted value as excluded, old/existing by the target table name and column:
upsert_sql = r"""insert into upsert_test_table
(id,encrypt_user_id,name)
values
""" + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT
"upsert_test_table_pkey"
DO UPDATE SET
name = upsert_test_table.name, --keep the old name
encrypt_user_id = excluded.encrypt_user_id --overwrite with new
"""
答案2
得分: -1
I apologize for any misunderstanding earlier. Here is the translated portion of the text:
抱歉,我刚才理解错了,我以为excluded表是原始(输入)表,但当我用excluded.user_id替换为excluded.encrypt_user_id时,它起作用了,尽管我不知道excluded表是如何工作的。
upsert_sql = r""""insert into upsert_test_table
(id,encrypt_user_id,name)
values
""" + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT
"upsert_test_table_pkey"
DO UPDATE SET
name=excluded.name,
encrypt_user_id=excluded.encrypt_user_id""""
英文:
Solved:
Sorry I was stupid, I thought excluded table is original(input payload) table, but when I replaced excluded.user_id with excluded.encrypt_user_id and it works, although have no idea how excluded table works.
upsert_sql = r"""insert into upsert_test_table
(id,encrypt_user_id,name)
values
""" + "\n, ".join(values_list) + """ ON CONFLICT ON CONSTRAINT
"upsert_test_table_pkey"
DO UPDATE SET
name=excluded.name,
encrypt_user_id=excluded.encrypt_user_id"""
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。


评论