Add jdbc-v2: read DatabaseMetaData with SHOW and DESCRIBE statements - #3168
Merged
Merged
Conversation
getSchemas(), getTables() and getColumns() now use SHOW DATABASES, SHOW FULL TABLES and DESCRIBE TABLE instead of system.databases, system.tables and system.columns. SHOW TABLES FROM <db> also lists the tables that the server hides from system.tables by default (tables of DataLakeCatalog and remote databases). The new driver property jdbc_metadata_use_show_statements (default true) selects the implementation; false keeps the system table queries. The result sets have the same columns and values, except that getTables() returns null in REMARKS and TYPE_SCHEM. getColumns() skips a table that fails with UNKNOWN_TABLE, UNKNOWN_DATABASE, ACCESS_DENIED or DATALAKE_DATABASE_ERROR. Implements: #2907 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Client V2 CoverageCoverage Report
Class Coverage
|
JDBC V2 CoverageCoverage Report
Class Coverage
|
JDBC V1 CoverageCoverage Report
Class Coverage
|
Client V1 CoverageCoverage Report
Class Coverage
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
1 task
- Document that the SHOW path returns rows in the JDBC order: the server sorts SHOW DATABASES / SHOW TABLES by name and DESCRIBE returns the columns in table order. Add a test for the order of getSchemas() with the flag on and off. - Mark the two methods that build SHOW / DESCRIBE queries with @SuppressWarnings("squid:S2077"), as getColumns() does: the queries contain only values escaped by likeLiteral() and quoteIdentifier(). Add parity cases with quotes in the patterns, including "%' OR '1'='1". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TriageCategory: Summary What this impacts
Concerns
Required reviewer action
|
chernser
requested changes
Sep 30, 2026
chernser
left a comment
chernser
left a comment
Contributor
There was a problem hiding this comment.
new code is a massive change and later it will be very hard to maintain it. Thus we need to move new logic into smaller internal class, use aggregation in existing to create instance of new logic class and just call it. At the same time old implementation should be moved to separate private methods of existing class and it will be a nice fork
public ResultSet getSchemas() throws SQLException {
if (use_show_statements) {
return sqlDataBaseMetadataImpl.getSchemas();
} else {
return getSchemasimpl();
}
}
This might also help to reuse some code.
Another issue - you are using statements but must use prepared statements
Fix all this, please.
Move the SHOW DATABASES / SHOW TABLES / DESCRIBE TABLE logic out of DatabaseMetaDataImpl into the internal class ShowStatementsMetaData. DatabaseMetaDataImpl creates one instance and forks in getSchemas(), getSchemas(String, String), getTables() and getColumns(): the flag selects ShowStatementsMetaData or the unchanged system table query, which is now in a private *Impl method. The SHOW statements are prepared statements. The LIKE patterns are parameters. The server accepts only an identifier after FROM and DESCRIBE TABLE, so database and table names stay escaped identifiers. Implements: #2907 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
@chernser thank you for the review. I pushed f1a948e with these changes: 1. New logic in a separate internal class
2. Prepared statements
Tests: |
…a-show-statements # Conflicts: # jdbc-v2/src/main/java/com/clickhouse/jdbc/metadata/DatabaseMetaDataImpl.java
chernser
requested changes
Oct 1, 2026
chernser
left a comment
chernser
left a comment
Contributor
There was a problem hiding this comment.
Implementation looks good. But tests should be changed:
- create ShowStatementDatabaseMetaDataTest that extends DatabaseMetadataTest but sets configuration flag to enable use of show statements medatata. The old tests should set flag to disable new show statement
- move jdbc-v2/src/test/java/com/clickhouse/jdbc/metadata/ShowStatementsMetaDataLikeMatcherTest.java to that new tests classes and make it interact with database.
…class DatabaseMetaDataTest now disables jdbc_metadata_use_show_statements, so it covers the system table queries. ShowStatementDatabaseMetaDataTest extends it and enables the flag, so the whole suite also runs on the SHOW/DESCRIBE path. The show-vs-system parity, flag and readonly user tests move to the new class. ShowStatementDatabaseMetaDataWithEmptyFormatTest runs the SHOW suite with an empty format, the same as DatabaseMetaDataWithEmptyFormatTest does for the system tables. The LIKE matcher unit test becomes a database test: it checks each case with the server's LIKE and through getColumns(). likeMatcher() is now private. testGetColumnsOfTemporaryTable covers the schema patterns that select temporary tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
@chernser Thank you. I changed the tests in 05ef73c:
Mutation check: when the SHOW path is broken, only the new classes fail (inherited Tests: the 4 metadata test classes, 290 run, 0 failures, 21 skipped (the empty-format skips), on ClickHouse 25.8 and 26.9. |
|
chernser
approved these changes
Oct 1, 2026
Contributor
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 05ef73c. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #2907 (as requested by @chernser in #2907 (comment)).
DatabaseMetaData#getSchemas(),#getSchemas(String, String),#getTablesand#getColumnscan now read metadata withSHOW DATABASES,SHOW FULL TABLESandDESCRIBE TABLEinstead ofsystem.databases,system.tablesandsystem.columns. A new driver property selects the implementation:jdbc_metadata_use_show_statements(DriverProperties.METADATA_USE_SHOW_STATEMENTS), defaulttrue. Set it tofalseto use the system tables as before.Why: the server does not show some databases and tables in the
system.*tables by default.SHOW TABLES FROM <db>turns onshow_data_lake_catalogs_in_system_tables/show_remote_databases_in_system_tablesfor the requested database (seeInterpreterShowTablesQuery::executein the server), but a plainSELECT … FROM system.tablesdoes not. So tables ofDataLakeCatalog(and remote) databases are listed only by theSHOWpath.Design
…WithShowStatementsimplementation. The legacysystem.*code is not changed and runs when the flag isfalse.DetachedResultSet#createFromRecords) with the same column names, JDBC types and values as thesystem.*queries:TABLE_TYPEuses the existing engine mapping (TABLE_TYPE_MUTATOR). The type filter keeps the existing behavior, including "only unknown types requested ⇒ all non-temporary tables".COLUMN_SIZE,DECIMAL_DIGITS,NUM_PREC_RADIXandCHAR_OCTET_LENGTHare computed from the type the same waysystem.columnsfillsnumeric_precision/numeric_precision_radix/numeric_scale/character_octet_length: onlyNullable(and theSimpleAggregateFunctionwrapper) is unwrapped; integers (incl.Bool) give their bit width, decimals their precision and scale,FixedStringits length.DATA_TYPEuses the existing mutator.getColumns()includes temporary tables (SHOW FULL TEMPORARY TABLES, emptyTABLE_SCHEM) likesystem.columns.getTables()does not, like the legacy query (itsJOIN system.databasesdrops them).LIKEpatterns are escaped like aPreparedStatementstring parameter (the legacygetTablesbinds them withsetString). Identifiers are back-quoted with\and`escaped.UNKNOWN_TABLE(60),UNKNOWN_DATABASE(81),ACCESS_DENIED(497) orDATALAKE_DATABASE_ERROR(736) is skipped (debug log), so one object does not fail the whole call. Examples: the object was dropped betweenSHOWandDESCRIBE; a user with only column-level grants, whereSHOW TABLESlists the table butDESCRIBEneedsSHOW COLUMNSon the whole table (system.columnsshows only the granted columns); a data lake table whose metadata cannot be read. Other errors are still thrown.docs/features.mdand the property javadoc):getTables()returnsnullinREMARKSandTYPE_SCHEM.SHOW TABLESreturns no table comment and no database engine. In the SHOW-path result set these two columns areNullable(String); the JDBC type is stillVARCHAR.getColumns()sends oneDESCRIBE TABLEper matching table (plus oneSHOW DATABASES, oneSHOW FULL TABLESper matching database and, when the schema pattern matches'', oneSHOW FULL TEMPORARY TABLES). A wide pattern such asgetColumns(null, null, "%", null)therefore sends many more queries than the singlesystem.columnsquery.getColumns()(see above).DESCRIBEsettings are not changed in the query. Areadonly=1user getsCannot modify 'print_pretty_type_names' setting in readonly modefrom aSETTINGSclause. Instead, the multi-line named-tuple type names thatprint_pretty_type_names=1(the server default) produces are normalized to thesystem.columnsform. Rows withis_subcolumn/is_virtual= 1 are skipped, and missingdefault_expression/commentcolumns (describe_compact_output) are tolerated.SHOW … LIKE ''is treated by the server as "no filter", butLIKE ''matches only an empty name. Empty schema and table patterns therefore return nothing, the same as thesystem.*queries.DESCRIBEcannot be a subquery, so the column-name pattern is matched on the client with ClickHouseLIKEsemantics (%,_,\escape; case-sensitive). The expected values in the unit test were checked against the server'sLIKE.Changes
DriverProperties: newMETADATA_USE_SHOW_STATEMENTSconstant (appended).DatabaseMetaDataImpl: flag dispatch ingetSchemas×2 /getTables/getColumns, plus the SHOW/DESCRIBE implementation.DetachedResultSet:createFromRecordsfactory for client-built rows.DatabaseMetaDataTest(parity + flag tests; they also run inDatabaseMetaDataWithEmptyFormatTest), newDatabaseMetaDataImplLikeMatcherTest(unit).CHANGELOG.md,docs/features.md.Test
DatabaseMetaDataTest/DatabaseMetaDataWithEmptyFormatTesttests now run on the SHOW path (default) and pass unchanged.testShowStatementsReturnSameMetadataAsSystemTables(@DataProvider, 14 cases): runs the samegetColumns/getTables/getSchemascall on a default connection and on ajdbc_metadata_use_show_statements=falseconnection. It asserts equal column labels, JDBC types, type names and nullability, and equal rows (excluding onlyREMARKS/TYPE_SCHEMforgetTables). It also asserts that rows exist where they must. The fixture has a wide type set (integers incl.Int128/UInt256,Bool,Nullable(Bool), decimals,FixedString,Nullable(FixedString),LowCardinality(FixedString), nested named tuples,Map,JSON(a.b UInt32),SimpleAggregateFunction,DEFAULT/MATERIALIZED/ALIAS/EPHEMERALcolumns, column comments), a view, a materialized view and a table whose table and column names contain a space,.,',",`and\. Patterns include a column prefix, escaped\_in column/table patterns, a schema pattern with a wildcard, empty patterns,VIEW/MATERIALIZED VIEWand unknown type filters, andsystemtables. A mutation check (removeBoolfrom the integer set, disable the pretty-name normalization) makes this test fail.testTableRemarksDependOnShowStatementsFlag(@DataProvider): pins the default (true) and both explicit values.testGetColumnsForReadonlyUserWithColumnGrants(@DataProvider, skipped on cloud likeReadonlyProfileTest): areadonly=1user with a full grant on one table and a column-level grant on another. The SHOW path works forreadonly=1, returns the normalized named-tuple type, and skips the table it cannot describe; the legacy path returns the granted column. The user has a random password and is dropped infinally. Grants are on the test database only.DatabaseMetaDataImplLikeMatcherTest(unit):LIKEedge cases (null/empty pattern,%,_, escapes, regex metacharacters, case, newline, surrogate pair).jdbc-v2verify(1952 unit + 574 integration tests). The only failures were the two HTTPS tests inConnectionTest(testSecureConnection,testSSLModeVerifyCa), which fail with "SSL Problem" because the local stack has no HTTPS endpoint. They are not related to this change.packages/clickhouse-jdbc-allbuilds.Invalid LZ4 magic byteissue ([client-v2] Every compressed read fails on ClickHouse 26.9+: response reader is hardcoded to LZ4 but the server default codec is now ZSTD(3) #3105/[client-v1] Every compressed read fails on ClickHouse 26.9+: Lz4InputStream hardcodes the LZ4 magic byte but the server default codec is now ZSTD #3107), which is not related to this change.DataLakeCatalogdatabase needs an external catalog, so CI cannot create one. Its visibility follows from the server code referenced above.Size
About 320 lines in
DatabaseMetaDataImpl, mostly the result-set column definitions and the per-row mapping needed for parity. Tests add about 240 lines.Possible follow-ups (not in this PR)
system.columnsfor ordinary databases andDESCRIBEonly for data lake / remote databases, to avoid the per-table queries.TYPE_SCHEMfrom a database-engine lookup. This PR leaves itnullto avoidsystem.*reads, as requested.Compatibility
getTables()REMARKS/TYPE_SCHEMbecomenull, andgetColumns()sends more queries. All other values are the same (see the parity test).jdbc_metadata_use_show_statements=falserestores the previous behavior exactly. The maintainer may prefer to list this entry under "Breaking Changes" in the CHANGELOG.DriverProperties.changes_checklist.md
jdbc_metadata_use_show_statementsis unique and follows thejdbc_*naming. Boolean, defaulttrue, parsed withJdbcConfiguration#isFlagSetlike the sibling flags. Documented in the javadoc anddocs/features.md. Focused tests added.getTables()REMARKS/TYPE_SCHEM(see Compatibility). All othergetSchemas/getTables/getColumnsoutput is pinned equal by the parity test.DriverProperties. No switch statements use this enum.DetachedResultSet#createFromRecordsis in the internal package, which is documented as not for application use.Pre-PR validation gate
Connection#getMetaData()→getSchemas/getTables/getColumns)DriverPropertiesflagssystem.*path; mutation-checked); no existing test was editedCHANGELOG.mdanddocs/features.mdupdatedAGENTS.md/docs/ai-review.md/docs/changes_checklist.md🤖 Generated with Claude Code