Skip to content

feat: add MySQL provider (Thinktecture.EntityFrameworkCore.MySql) - #77

Open
pengweiqhca wants to merge 2 commits into
PawelGerr:masterfrom
pengweiqhca:mysql
Open

pengweiqhca wants to merge 2 commits into
PawelGerr:masterfrom
pengweiqhca:mysql

Conversation

@pengweiqhca

Copy link
Copy Markdown

The new package builds on Microting.EntityFrameworkCore.MySql, the Pomelo fork for EF Core 10, and brings the features of the other providers to MySQL and MariaDB: bulk insert, update and upsert, bulk insert and update from a query, temp tables, truncate, collection parameters, window functions, bitwise aggregates and nested transactions. It is registered with AddBulkOperationSupport, AddWindowFunctionsSupport and AddCollectionParameterSupport on the MySqlDbContextOptionsBuilder. Window functions require MySQL 8.0 (MariaDB 10.2) and collection parameters require MySQL 8.0.4 (MariaDB 10.6); the configured server version is validated and an InvalidOperationException is thrown otherwise. Table hints, tenant databases and the migration helpers are not available.

Bulk insert writes multi-row INSERT statements (BatchSize, default 500, capped by the limit of 65,535 parameters; CommandTimeout) and creates the parameters through the type mapping of each column. Without an outer transaction the executor opens its own, so a failing batch leaves nothing behind. UseBulkCopy opts into MySqlBulkCopy (LOAD DATA LOCAL INFILE) and fails on warnings such as dropped duplicate keys. Owned types in separate tables are supported for insert. Bulk update stages the rows in a temp table and runs one UPDATE with a JOIN, or updates row by row inside a TransactionScope because MySQL rejects temp tables in XA transactions. Bulk upsert uses INSERT ... ON DUPLICATE KEY UPDATE. MySQL fires the clause for every unique index, so each assignment is guarded by the key properties: key columns are never updated and a row that collides with a different row on another unique index is skipped. The result is the number of processed input rows, because the affected-row counts of MySQL differ for inserted, updated and unchanged rows. MySQL 8.0.19 and later use the row alias instead of the deprecated VALUES(col). Truncate uses DELETE FROM inside a transaction (TRUNCATE commits implicitly) and for tables that are referenced by a foreign key (error 1701).

Temp tables are connection-scoped and carry the character set and collation of the model for text columns only, because MySQL rejects them for json columns and fails joins with error 1267 otherwise. MySQL cannot reference a temp table twice in one statement (error 1137). Collection parameters are read with JSON_TABLE. MySQL parses JSON numbers as DOUBLE and rejects the .NET formats of TimeSpan and DateTime, so decimal, TimeSpan and DateTime are serialized as strings, and converters that EF Core infers from the provider type are applied. The SQL of window functions has no space before the opening parenthesis, ROW_NUMBER is cast to signed, and the bitwise aggregates cast to SIGNED and return NULL for a group without values.

The tests run against a pinned mysql:8.4 container (Testcontainers) and use the shared entities of Thinktecture.EntityFrameworkCore.TestHelpers together with the three-context pattern and command capturing of the other providers. Known problems of the shared translator and of the Pomelo SQL helper are documented by skipped tests: the table name override of query-based updates is a textual replace, DateTime constants lose their fractional seconds in an UPDATE, and the schema of IBulkOperationTableNameProvider.ForRootEntity is ignored. README and CLAUDE.md describe the provider; the wiki and the agent skill are updated separately.

…s the isolation level Unspecified

Some providers do not report the isolation level of a transaction that was started without an explicit level. MySqlConnector returns IsolationLevel.Unspecified in this case. The nested transaction compared the requested level of a child transaction against that value, so every child transaction with an explicit isolation level failed with "The isolation level 'Unspecified' of the parent transaction is not compatible", although the root transaction ran with the default of the server.

A parent that reports Unspecified is now treated as REPEATABLE READ, which is the default of MySQL and MariaDB. Child transactions that request REPEATABLE READ or a weaker level are accepted, stricter levels are still rejected. Parents that report a concrete level behave as before.
The new package builds on Microting.EntityFrameworkCore.MySql, the Pomelo fork for EF Core 10, and brings the features of the other providers to MySQL and MariaDB: bulk insert, update and upsert, bulk insert and update from a query, temp tables, truncate, collection parameters, window functions, bitwise aggregates and nested transactions. It is registered with AddBulkOperationSupport, AddWindowFunctionsSupport and AddCollectionParameterSupport on the MySqlDbContextOptionsBuilder. Window functions require MySQL 8.0 (MariaDB 10.2) and collection parameters require MySQL 8.0.4 (MariaDB 10.6); the configured server version is validated and an InvalidOperationException is thrown otherwise. Table hints, tenant databases and the migration helpers are not available.

Bulk insert writes multi-row INSERT statements (BatchSize, default 500, capped by the limit of 65,535 parameters; CommandTimeout) and creates the parameters through the type mapping of each column. Without an outer transaction the executor opens its own, so a failing batch leaves nothing behind. UseBulkCopy opts into MySqlBulkCopy (LOAD DATA LOCAL INFILE) and fails on warnings such as dropped duplicate keys. Owned types in separate tables are supported for insert. Bulk update stages the rows in a temp table and runs one UPDATE with a JOIN, or updates row by row inside a TransactionScope because MySQL rejects temp tables in XA transactions. Bulk upsert uses INSERT ... ON DUPLICATE KEY UPDATE. MySQL fires the clause for every unique index, so each assignment is guarded by the key properties: key columns are never updated and a row that collides with a different row on another unique index is skipped. The result is the number of processed input rows, because the affected-row counts of MySQL differ for inserted, updated and unchanged rows. MySQL 8.0.19 and later use the row alias instead of the deprecated VALUES(col). Truncate uses DELETE FROM inside a transaction (TRUNCATE commits implicitly) and for tables that are referenced by a foreign key (error 1701).

Temp tables are connection-scoped and carry the character set and collation of the model for text columns only, because MySQL rejects them for json columns and fails joins with error 1267 otherwise. MySQL cannot reference a temp table twice in one statement (error 1137). Collection parameters are read with JSON_TABLE. MySQL parses JSON numbers as DOUBLE and rejects the .NET formats of TimeSpan and DateTime, so decimal, TimeSpan and DateTime are serialized as strings, and converters that EF Core infers from the provider type are applied. The SQL of window functions has no space before the opening parenthesis, ROW_NUMBER is cast to signed, and the bitwise aggregates cast to SIGNED and return NULL for a group without values.

The tests run against a pinned mysql:8.4 container (Testcontainers) and use the shared entities of Thinktecture.EntityFrameworkCore.TestHelpers together with the three-context pattern and command capturing of the other providers. Known problems of the shared translator and of the Pomelo SQL helper are documented by skipped tests: the table name override of query-based updates is a textual replace, DateTime constants lose their fractional seconds in an UPDATE, and the schema of IBulkOperationTableNameProvider.ForRootEntity is ignored. README and CLAUDE.md describe the provider; the wiki and the agent skill are updated separately.
@PawelGerr

Copy link
Copy Markdown
Owner

Hi @pengweiqhca,
thank you very much for this contribution. A MySQL and MariaDB provider is a great addition, and the PR description is very detailed, which helps a lot. The PR is large and touches many features, so I need some time to review it properly. I will come back to you with feedback as soon as I can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants