最佳实践:在生产环境中删除和更新主/唯一键而不产生停机时间。

huangapple go评论50阅读模式
英文:

Best practices to drop & update primary/unique keys on production without having a downtime

问题

我正在实施MySQL服务器(8.0)上的数据分区。

作为“分区的必备要求”,我必须将我的分区键添加为主键,这在我的表中是created_at,例如:

alter table notifications drop primary key, add primary key(`id`, `created_at`);

在上面的查询中,id 是一个自增的主键。我的表中有超过1000万条记录。
当我运行alter命令时,MySQL不允许对模式进行实时更新;它会锁定表格。

同时,我可以为此过程停机。请指导采取最佳方法来进行此实时操作。

我尝试使用“Faker”库在本地生成相同数量的数据,并在其上运行alter查询。我观察到这个过程需要超过2小时。

Schema

CREATE TABLE `data` (
  `seq_id` int NOT NULL AUTO_INCREMENT,
  `id` varchar(100) NOT NULL,
  `doc_id` varchar(100) NOT NULL,
  `page_no` int DEFAULT '1',
  `file_store` varchar(100) DEFAULT NULL,
  `s_id` varchar(100) NOT NULL,
  `s_f_id` varchar(100) NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `f_store` varchar(100) DEFAULT 'GCP',
  `purged` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`seq_id`),
  UNIQUE KEY `id` (`id`),
  KEY `doc_id` (`doc_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

ALTER TABLE `data `
PARTITION BY RANGE (UNIX_TIMESTAMP(created_at))
(
  PARTITION `partition_2020` VALUES LESS THAN (UNIX_TIMESTAMP('2020-01-01 00:00:00')),
  PARTITION `partition_2021` VALUES LESS THAN (UNIX_TIMESTAMP('2021-01-01 00:00:00')), 
  PARTITION `partition_2022` VALUES LESS THAN (UNIX_TIMESTAMP('2022-01-01 00:00:00')), 
  PARTITION `partition_2022` VALUES LESS THAN (UNIX_TIMESTAMP('2023-05-31 00:00:00')),
  PARTITION `partition_future` VALUES LESS THAN (MAXVALUE)
);

英文:

I am implementing data partitioning on mysql server(8.0).

As a "must-have requirement of partitioning" I have to add my partition key as a primary key, which is created_at in my table. eg:

alter table notifications drop primary key, add primary key(`id`, `created_at`);

In above query, id is an auto-increment primary key. I have more than 10M entries in table.
Mysql won't allow live updates on schemas; it will lock the table when I run alter command.

At the same time, I can take a downtime for this process. Please guide the best approach to take this live.

I tried generating same amount of data on my local using "Faker" library & while running alter query on it. I observed it taking more than 2 hours for this process.

Schema

CREATE TABLE `data` (
  `seq_id` int NOT NULL AUTO_INCREMENT,
  `id` varchar(100) NOT NULL,
  `doc_id` varchar(100) NOT NULL,
  `page_no` int DEFAULT '1',
  `file_store` varchar(100) DEFAULT NULL,
  `s_id` varchar(100) NOT NULL,
  `s_f_id` varchar(100) NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `f_store` varchar(100) DEFAULT 'GCP',
  `purged` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`seq_id`),
  UNIQUE KEY `id` (`id`),
  KEY `doc_id` (`doc_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

ALTER TABLE `data `
PARTITION BY RANGE (UNIX_TIMESTAMP(created_at))
(
  PARTITION `partition_2020` VALUES LESS THAN (UNIX_TIMESTAMP('2020-01-01 00:00:00')),
  PARTITION `partition_2021` VALUES LESS THAN (UNIX_TIMESTAMP('2021-01-01 00:00:00')), 
  PARTITION `partition_2022` VALUES LESS THAN (UNIX_TIMESTAMP('2022-01-01 00:00:00')), 
  PARTITION `partition_2022` VALUES LESS THAN (UNIX_TIMESTAMP('2023-05-31 00:00:00')),
  PARTITION `partition_future` VALUES LESS THAN (MAXVALUE)
);

答案1

得分: 2

如果您非常关心锁定时间,那么我建议您使用主从复制。基本上,这是主从切换时间,非常短。

英文:

If you are very concerned about the lock time, then I suggest you use the master-slave replication. Basically, it is a master-slave switching time, which is very short.

答案2

得分: 0

Both changing the PK and adding PARTITIONing require a full copy and re-index of the data. This will be slow. But... Let's talk about whether the Partitioning will provide any benefit. Please provide SHOW CREATE TABLE -- both currently and after adding partitioning. See also Partition

If, on the other hand, you decide that Partitioning is really needed, and the table is growing, then the sooner you do it, the less time it will take.

If you will be purging some old data as you do the conversion, then I suggest doing everything at once:

CREATE TABLE new_t (
... ); -- with new PK and partitioning.
INSERT INTO new_t
SELECT * FROM t
WHERE date >= ...; -- to purge old data by not copying it over

But even before doing that, let's see the CREATEs, there may be more advice to fold into the copy.

Also note that Partitioning does not allow for UNIQUE keys, nor for FOREIGN KEYs. We can discuss that further after seeing the CREATE.

Also -- Do you have a "natural" PK? That is, can you get rid of id?

More

While changing the PK, can you simply remove the seq_id (auto_inc) column? Or do you need it for some other reason?

英文:

Both changing the PK and adding PARTITIONing require a full copy and re-index of the data. This will be slow. But... Let's talk about whether the Partitioning will provide any benefit. Please provide SHOW CREATE TABLE -- both currently and after adding partitioning. See also Partition

If, on the other hand, you decide that Partitioning is really needed, and the table is growing, then the sooner you do it, the less time it will take.

If you will be purging some old data as you do the conversion, then I suggest doing everything at once:

CREATE TABLE new_t (
    ... ); -- with new PK and partitioning.
INSERT INTO new_t
    SELECT * FROM t
        WHERE date >= ...; -- to purge old data by not copying it over

But even before doing that, let's see the CREATEs, there may be more advice to fold into the copy.

Also note that Partitioning does not allow for UNIQUE keys, nor for FOREIGN KEYs. We can discuss that further after seeing the CREATE.

Also -- Do you have a "natural" PK? That is, can you get rid of id?

More

While changing the PK, can you simply remove the seq_id (auto_inc) column? Or do you need it for some other reason?

huangapple
  • 本文由 发表于 2023年6月26日 13:53:51
  • 转载请务必保留本文链接:https://go.coder-hub.com/76553848.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定