diff --git a/CHANGELOG.md b/CHANGELOG.md index a4f830043..6717703f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - SSL settings on mobile for MySQL, PostgreSQL and Redis: a mode picker plus CA, client certificate and client key. Import a PEM file, paste one, or use a PKCS#12 file. (#2083) - Legacy UUID Encoding on a MongoDB connection, so binary UUIDs written by the Java, C# or Python drivers read as UUIDs instead of hex. Filters, edits and MQL exports write the same bytes back. (#2086) +### Added + +- The MongoDB editor accepts mongosh value constructors in filters and pipelines, so a value copied from the grid pastes straight into a query. Covers `ObjectId`, `ISODate`, `Date`, the `Number*` family, `Timestamp`, `BinData`, `HexData`, `MinKey`, `MaxKey` and the UUID names. (#2086) + ### Changed - Mobile keeps remote connections open when you switch apps. @@ -40,6 +44,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- MQL export keeps a binary value's BSON subtype and writes it as a `BinData(...)` constructor, so running the script inserts the same bytes back. It wrote Extended JSON that mongosh reads as a plain object, and stamped every value as subtype 0. (#2086) - MongoDB no longer prints a binary field nested inside a document as a UUID when it is not one, or labels a UUID with the wrong byte order. (#2086) - Deleting a MongoDB document with a binary `_id` deletes that document. It used to match on the other fields and could remove the wrong one. - TablePro Mobile no longer gets killed by iOS when you leave the app with a DuckDB file open. diff --git a/CLAUDE.md b/CLAUDE.md index 27ffa7c60..c3efc65d6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -199,7 +199,7 @@ These have caused real bugs when violated: **The data grid header owns all of its own chrome, so nothing may ask AppKit to paint any of it**: `NSTableHeaderCell` and `NSTableHeaderView` both paint a fixed 28pt band that they centre vertically in whatever frame they are given, a 16pt column divider on `midY` and a 1pt rule at `midY + 13`. The data grid grows its header to 42pt for a column comment, so that band lands mid-cell: the rule crosses the comment's descenders and sits 8pt above the real bottom edge. `SortableHeaderChrome` is therefore the single owner of header geometry and colours, `SortableHeaderCell.draw(withFrame:in:)` never calls `super`, and `SortableHeaderView.draw(_:)` fills the background and rules the bottom edge itself. The trap is that the header view paints a second copy of that same band for `NSTableView.highlightedTableColumn`, driven by *state* rather than by a drawing call, so no cell override can reach it: setting it gives the sorted column a stray divider and a rule no other column has. TablePro already draws the sorted-column affordance itself (bold title, chevron, priority number, with `drawSortIndicator` overridden to nothing), so `highlightedTableColumn` is a redundant second channel and must stay unset. All sorted-column presentation goes through `SortableHeaderView.applySortState(_:schema:)`, which publishes the order natively through `tableView.sortDescriptors` (for accessibility; it paints nothing) and updates the cells. `SortableHeaderRenderingTests` rasterises the header and guards this. This shipped as a rule through the comment line and a stray divider on the sorted column (#2017). -**Decoding a MongoDB binary UUID is a per-column decision, and the column's type name is load-bearing**: BSON binary subtype 3 is the legacy UUID format, and the Java, C# and Python drivers each wrote it with a different byte order with nothing in the stored bytes to say which. `MongoDBUuidCodec` therefore decodes subtype 3 only when the connection names one (`mongoUuidRepresentation`); subtype 4 is unambiguous and always decodes. The choice is made once per column from `BsonDocumentFlattener.columnKinds`' majority vote, never per value, because a decoded cell is `.text` and an undecoded one is `.bytes`, and `CellDisplayFormatter` runs blob formatting over a `.text` cell whenever its column type is BLOB. One UUID decoded inside a column the app still types `BLOB` renders as `0x4c65676163...`. For the same reason `BsonDocumentFlattener.typeName` must return exactly `"BLOB"` for undecoded binary: `ColumnTypeClassifier` maps any name containing `BLOB` to `.blob`, and that classification is the only thing keeping a binary cell out of the inline editor. Once a column does decode, both edit guards (`isBlobType` and `asBytes != nil`) fall together, so every write path must parse the wrapper back to `$binary`: `MongoDBStatementGenerator.jsonValue` and `idValueJson`, `MongoDBQueryBuilder.jsonValue` plus its `=`, `!=` and `IN` arms (a case-insensitive regex can never match a binary field), and `MQLExportHelpers.mqlJsonValue`. An `_id` filter left as wrapper text matches zero documents while the UI reports the save succeeded. (#2086) +**Decoding a MongoDB binary UUID is a per-column decision, and the column's type name is load-bearing**: BSON binary subtype 3 is the legacy UUID format, and the Java, C# and Python drivers each wrote it with a different byte order with nothing in the stored bytes to say which. `MongoDBUuidCodec` therefore decodes subtype 3 only when the connection names one (`mongoUuidRepresentation`); subtype 4 is unambiguous and always decodes. The choice is made once per column from `BsonDocumentFlattener.columnKinds`' majority vote, never per value, because a decoded cell is `.text` and an undecoded one is `.bytes`, and `CellDisplayFormatter` runs blob formatting over a `.text` cell whenever its column type is BLOB. One UUID decoded inside a column the app still types `BLOB` renders as `0x4c65676163...`. For the same reason `BsonDocumentFlattener.typeName` must keep `BLOB` as the base name for undecoded binary: `ColumnTypeClassifier` splits a type name at the first `(` and looks the base up, so `BLOB` and `BLOB(3)` both classify as `.blob`, and that classification is the only thing keeping a binary cell out of the inline editor. The parenthesised part carries the BSON subtype so MQL export can write it back; `MongoDBUuidCodec.columnTypeName(forSubtype:)` and `binarySubtype(fromColumnTypeName:)` are the only two places that spelling is produced or read, and MQL export is `supportedDatabaseTypeIds = ["MongoDB"]`, so it never sees another driver's `BLOB`. Once a column does decode, both edit guards (`isBlobType` and `asBytes != nil`) fall together, so every write path must parse the wrapper back to `$binary`: `MongoDBStatementGenerator.jsonValue` and `idValueJson`, `MongoDBQueryBuilder.jsonValue` plus its `=`, `!=` and `IN` arms (a case-insensitive regex can never match a binary field), and `MQLExportHelpers.mqlJsonValue`. An `_id` filter left as wrapper text matches zero documents while the UI reports the save succeeded. (#2086) **A MongoDB update or delete is anchored on `_id` or it does not run**: `generateDelete` used to fall back to a filter built from the remaining columns, which silently dropped every value it could not stringify (all binary) and then `deleteOne`d the first partial match, so a document with a binary `_id` could delete a different document. Both paths now skip with a logged warning instead, matching what `generateUpdate` already did. diff --git a/Plugins/MQLExportPlugin/MQLExportHelpers.swift b/Plugins/MQLExportPlugin/MQLExportHelpers.swift index 9bf31ccf0..9a4880f4b 100644 --- a/Plugins/MQLExportPlugin/MQLExportHelpers.swift +++ b/Plugins/MQLExportPlugin/MQLExportHelpers.swift @@ -24,6 +24,10 @@ enum MQLExportHelpers { return "db.\(escaped)" } + static func mqlBinaryValue(for data: Data, subtype: UInt8) -> String { + MongoDBUuidCodec.binaryText(for: MongoDBBinaryValue(data: data, subtype: subtype)) + } + static func mqlJsonValue(for value: String) -> String { if value == "true" || value == "false" { return value @@ -37,8 +41,8 @@ enum MQLExportHelpers { if Double(value) != nil, value.contains(".") { return value } - if let binary = MongoDBUuidCodec.extendedJsonFromWrapper(value) { - return binary + if let binary = MongoDBUuidCodec.parseWrapper(value) { + return MongoDBUuidCodec.binaryText(for: binary) } if (value.hasPrefix("{") && value.hasSuffix("}")) || (value.hasPrefix("[") && value.hasSuffix("]")) { diff --git a/Plugins/MQLExportPlugin/MQLExportPlugin.swift b/Plugins/MQLExportPlugin/MQLExportPlugin.swift index 90e19d6d8..9c62c36fd 100644 --- a/Plugins/MQLExportPlugin/MQLExportPlugin.swift +++ b/Plugins/MQLExportPlugin/MQLExportPlugin.swift @@ -94,6 +94,7 @@ final class MQLExportPlugin: ExportFormatPlugin, SettablePlugin { if includeData { var columns: [String] = [] + var columnTypeNames: [String] = [] var documentBatch: [String] = [] let stream = dataSource.streamRows(table: table.name, databaseName: table.databaseName) @@ -103,6 +104,7 @@ final class MQLExportPlugin: ExportFormatPlugin, SettablePlugin { switch element { case .header(let header): columns = header.columns + columnTypeNames = header.columnTypeNames case .rows(let rows): for row in rows { var fields: [String] = [] @@ -114,7 +116,11 @@ final class MQLExportPlugin: ExportFormatPlugin, SettablePlugin { case .null: continue case .bytes(let data): - jsonValue = "{\"$binary\": {\"base64\": \"\(data.base64EncodedString())\", \"subType\": \"00\"}}" + let typeName = colIndex < columnTypeNames.count ? columnTypeNames[colIndex] : "" + jsonValue = MQLExportHelpers.mqlBinaryValue( + for: data, + subtype: MongoDBUuidCodec.binarySubtype(fromColumnTypeName: typeName) + ) case .text(let value): jsonValue = MQLExportHelpers.mqlJsonValue(for: value) } diff --git a/Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift b/Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift index 6f5bf7ddd..af2feef76 100644 --- a/Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift +++ b/Plugins/MongoDBDriverPlugin/BsonDocumentFlattener.swift @@ -9,12 +9,12 @@ import Foundation import TableProPluginKit -enum BsonValueKind { +enum BsonValueKind: Hashable { case double case string case document case array - case binary + case binary(subtype: UInt8) case boolean case date case null @@ -113,7 +113,7 @@ struct BsonDocumentFlattener { case .double: return "FLOAT" case .string, .null: return "VARCHAR" case .document, .array: return "JSON" - case .binary: return "BLOB" + case .binary(let subtype): return MongoDBUuidCodec.columnTypeName(forSubtype: subtype) case .boolean: return "BOOLEAN" case .date: return "TIMESTAMP" case .int32: return "INTEGER" @@ -290,11 +290,11 @@ struct BsonDocumentFlattener { return .date case let binary as MongoDBBinaryValue: guard MongoDBUuidCodec.isDecodableUuid(binary, representation: representation) else { - return .binary + return .binary(subtype: binary.subtype) } return binary.subtype == MongoDBUuidCodec.standardUuidSubtype ? .uuid : .legacyUuid case is Data: - return .binary + return .binary(subtype: 0) case is [String: Any]: return .document case is [Any]: diff --git a/Plugins/MongoDBDriverPlugin/MongoDBPlugin.swift b/Plugins/MongoDBDriverPlugin/MongoDBPlugin.swift index 23570db40..3b49a6355 100644 --- a/Plugins/MongoDBDriverPlugin/MongoDBPlugin.swift +++ b/Plugins/MongoDBDriverPlugin/MongoDBPlugin.swift @@ -77,9 +77,9 @@ final class MongoDBPlugin: NSObject, TableProPlugin, DriverPlugin { label: String(localized: "Legacy UUID Encoding"), fieldType: .dropdown(options: [ .init(value: "", label: String(localized: "Do Not Decode")), - .init(value: "javaLegacy", label: String(localized: "Java")), - .init(value: "csharpLegacy", label: String(localized: "C#")), - .init(value: "pythonLegacy", label: String(localized: "Python")), + .init(value: "javaLegacy", label: "Java"), + .init(value: "csharpLegacy", label: "C#"), + .init(value: "pythonLegacy", label: "Python"), ]), section: .advanced ), @@ -141,7 +141,13 @@ final class MongoDBPlugin: NSObject, TableProPlugin, DriverPlugin { CompletionEntry(label: ".findOneAndReplace", insertText: ".findOneAndReplace"), CompletionEntry(label: ".findOneAndDelete", insertText: ".findOneAndDelete"), CompletionEntry(label: ".countDocuments", insertText: ".countDocuments"), - CompletionEntry(label: ".createIndex", insertText: ".createIndex") + CompletionEntry(label: ".createIndex", insertText: ".createIndex"), + CompletionEntry(label: "ObjectId", insertText: "ObjectId"), + CompletionEntry(label: "ISODate", insertText: "ISODate"), + CompletionEntry(label: "NumberLong", insertText: "NumberLong"), + CompletionEntry(label: "NumberDecimal", insertText: "NumberDecimal"), + CompletionEntry(label: "BinData", insertText: "BinData"), + CompletionEntry(label: "UUID", insertText: "UUID") ] } diff --git a/Plugins/TableProPluginKit/MongoDBUuidCodec.swift b/Plugins/TableProPluginKit/MongoDBUuidCodec.swift index 1e2adde8a..08f5db9d4 100644 --- a/Plugins/TableProPluginKit/MongoDBUuidCodec.swift +++ b/Plugins/TableProPluginKit/MongoDBUuidCodec.swift @@ -44,6 +44,19 @@ public enum MongoDBUuidCodec { "BinData(\(binary.subtype), \"\(binary.data.base64EncodedString())\")" } + public static func columnTypeName(forSubtype subtype: UInt8) -> String { + subtype == 0 ? binaryColumnTypeName : "\(binaryColumnTypeName)(\(subtype))" + } + + public static func binarySubtype(fromColumnTypeName name: String) -> UInt8 { + guard name.hasPrefix(binaryColumnTypeName + "("), name.hasSuffix(")") else { return 0 } + let start = name.index(name.startIndex, offsetBy: binaryColumnTypeName.count + 1) + let digits = name[start ..< name.index(before: name.endIndex)] + return UInt8(digits) ?? 0 + } + + private static let binaryColumnTypeName = "BLOB" + public static func isDecodableUuid( _ binary: MongoDBBinaryValue, representation: MongoDBUuidRepresentation diff --git a/Plugins/TableProPluginKit/MongoShellParser.swift b/Plugins/TableProPluginKit/MongoShellParser.swift index cd0ce80c6..9de93cf81 100644 --- a/Plugins/TableProPluginKit/MongoShellParser.swift +++ b/Plugins/TableProPluginKit/MongoShellParser.swift @@ -77,7 +77,8 @@ public struct MongoShellParser { /// Parse a MongoDB Shell expression into a MongoOperation public static func parse(_ input: String) throws -> MongoOperation { - let trimmed = input.trimmingCharacters(in: .whitespacesAndNewlines) + let trimmed = try MongoShellValueTranslator.translate(input) + .trimmingCharacters(in: .whitespacesAndNewlines) if trimmed.isEmpty { throw MongoShellParseError.invalidSyntax("Empty query") diff --git a/Plugins/TableProPluginKit/MongoShellValueTranslator.swift b/Plugins/TableProPluginKit/MongoShellValueTranslator.swift new file mode 100644 index 000000000..116c6bda9 --- /dev/null +++ b/Plugins/TableProPluginKit/MongoShellValueTranslator.swift @@ -0,0 +1,348 @@ +import Foundation + +public enum MongoShellValueError: Error, LocalizedError, Equatable { + case malformedArguments(helper: String) + case unterminatedString + case unterminatedCall(helper: String) + + public var errorDescription: String? { + switch self { + case .malformedArguments(let helper): + return String(format: String(localized: "%@ was given arguments it cannot use."), helper) + case .unterminatedString: + return String(localized: "A string literal is missing its closing quote.") + case .unterminatedCall(let helper): + return String(format: String(localized: "%@ is missing its closing parenthesis."), helper) + } + } +} + +/// Rewrites the mongosh value constructors a user can type into the canonical Extended JSON +/// that libbson accepts. Identifiers preceded by a dot are method calls and are left alone, +/// which also makes the pass idempotent. +public enum MongoShellValueTranslator { + public static func translate(_ input: String) throws -> String { + guard containsCandidate(input) else { return input } + + var output = "" + output.reserveCapacity(input.count) + let characters = Array(input) + var index = 0 + var previousMeaningful: Character? + + while index < characters.count { + let character = characters[index] + + if character == "\"" || character == "'" { + output += try readStringLiteral(characters, from: &index) + previousMeaningful = "\"" + continue + } + + if isIdentifierStart(character), previousMeaningful != "." { + if let translated = try translateCall(characters, from: &index) { + output += translated + previousMeaningful = ")" + continue + } + } + + output.append(character) + if !character.isWhitespace { previousMeaningful = character } + index += 1 + } + + return output + } + + // MARK: - Call Translation + + private static func translateCall(_ characters: [Character], from index: inout Int) throws -> String? { + let start = index + var cursor = index + while cursor < characters.count, isIdentifierPart(characters[cursor]) { cursor += 1 } + let name = String(characters[start ..< cursor]) + + var afterName = cursor + while afterName < characters.count, characters[afterName].isWhitespace { afterName += 1 } + + guard afterName < characters.count, characters[afterName] == "(" else { + guard let bare = bareConstant(name) else { return nil } + index = cursor + return bare + } + guard isKnownHelper(name) else { return nil } + + guard let closing = matchingParenthesis(characters, openIndex: afterName) else { + throw MongoShellValueError.unterminatedCall(helper: name) + } + let arguments = try splitArguments(Array(characters[(afterName + 1) ..< closing])) + guard let json = try extendedJson(helper: name, arguments: arguments) else { return nil } + + index = closing + 1 + return json + } + + private static func extendedJson(helper: String, arguments: [String]) throws -> String? { + func malformed() -> MongoShellValueError { .malformedArguments(helper: helper) } + + switch helper { + case "ObjectId": + let hex = try singleStringArgument(arguments, helper: helper) + guard hex.count == 24, hex.allSatisfy({ $0.isHexDigit && $0.isASCII }) else { throw malformed() } + return "{\"$oid\": \"\(escaped(hex))\"}" + case "ISODate", "Date": + let value = try singleStringArgument(arguments, helper: helper) + return "{\"$date\": \"\(escaped(value))\"}" + case "NumberInt": + let raw = try numericArgument(arguments, helper: helper) + guard Int32(raw) != nil else { throw malformed() } + return "{\"$numberInt\": \"\(raw)\"}" + case "NumberLong": + let raw = try numericArgument(arguments, helper: helper) + guard Int64(raw) != nil else { throw malformed() } + return "{\"$numberLong\": \"\(raw)\"}" + case "NumberDecimal": + return "{\"$numberDecimal\": \"\(try numericArgument(arguments, helper: helper))\"}" + case "Timestamp": + guard arguments.count == 2, + let seconds = UInt32(unwrapped(arguments[0])), + let increment = UInt32(unwrapped(arguments[1])) else { throw malformed() } + return "{\"$timestamp\": {\"t\": \(seconds), \"i\": \(increment)}}" + case "BinData": + guard arguments.count == 2, let subtype = UInt8(unwrapped(arguments[0])), + let payload = stringLiteralValue(arguments[1]), + Data(base64Encoded: payload) != nil else { throw malformed() } + return MongoDBUuidCodec.extendedJson( + for: MongoDBBinaryValue(data: Data(base64Encoded: payload) ?? Data(), subtype: subtype) + ) + case "HexData": + guard arguments.count == 2, let subtype = UInt8(unwrapped(arguments[0])), + let payload = stringLiteralValue(arguments[1]), + let data = hexData(payload) else { throw malformed() } + return MongoDBUuidCodec.extendedJson(for: MongoDBBinaryValue(data: data, subtype: subtype)) + case "MinKey": + guard arguments.isEmpty else { throw malformed() } + return "{\"$minKey\": 1}" + case "MaxKey": + guard arguments.isEmpty else { throw malformed() } + return "{\"$maxKey\": 1}" + default: + let uuid = try singleStringArgument(arguments, helper: helper) + guard let binary = MongoDBUuidCodec.parseWrapper("\(helper)(\"\(uuid)\")") else { throw malformed() } + return MongoDBUuidCodec.extendedJson(for: binary) + } + } + + // MARK: - Helper Vocabulary + + private static let valueHelpers: Set = [ + "ObjectId", "ISODate", "Date", "NumberInt", "NumberLong", "NumberDecimal", + "Timestamp", "BinData", "HexData", "MinKey", "MaxKey", + ] + + private static let uuidHelpers: Set = [ + "UUID", "LegacyJavaUUID", "LegacyCSharpUUID", "LegacyPythonUUID", + "JUUID", "CSUUID", "NUUID", "PYUUID", "LUUID", + ] + + private static func isKnownHelper(_ name: String) -> Bool { + valueHelpers.contains(name) || uuidHelpers.contains(name) + } + + private static func bareConstant(_ name: String) -> String? { + switch name { + case "MinKey": return "{\"$minKey\": 1}" + case "MaxKey": return "{\"$maxKey\": 1}" + default: return nil + } + } + + private static func containsCandidate(_ input: String) -> Bool { + for helper in valueHelpers.union(uuidHelpers) where input.contains(helper) { + return true + } + return false + } + + // MARK: - Argument Parsing + + private static func singleStringArgument(_ arguments: [String], helper: String) throws -> String { + guard arguments.count == 1, let value = stringLiteralValue(arguments[0]) else { + throw MongoShellValueError.malformedArguments(helper: helper) + } + return value + } + + private static func numericArgument(_ arguments: [String], helper: String) throws -> String { + guard arguments.count == 1 else { throw MongoShellValueError.malformedArguments(helper: helper) } + let raw = stringLiteralValue(arguments[0]) ?? unwrapped(arguments[0]) + guard isNumericLiteral(raw) else { + throw MongoShellValueError.malformedArguments(helper: helper) + } + return raw + } + + private static func isNumericLiteral(_ value: String) -> Bool { + var characters = Array(value)[...] + if characters.first == "-" || characters.first == "+" { characters = characters.dropFirst() } + + var digits = 0 + while let first = characters.first, first.isNumber, first.isASCII { + characters = characters.dropFirst() + digits += 1 + } + if characters.first == "." { + characters = characters.dropFirst() + while let first = characters.first, first.isNumber, first.isASCII { + characters = characters.dropFirst() + digits += 1 + } + } + guard digits > 0 else { return false } + + if characters.first == "e" || characters.first == "E" { + characters = characters.dropFirst() + if characters.first == "-" || characters.first == "+" { characters = characters.dropFirst() } + var exponentDigits = 0 + while let first = characters.first, first.isNumber, first.isASCII { + characters = characters.dropFirst() + exponentDigits += 1 + } + guard exponentDigits > 0 else { return false } + } + return characters.isEmpty + } + + private static func stringLiteralValue(_ argument: String) -> String? { + let trimmed = unwrapped(argument) + guard trimmed.count >= 2, let first = trimmed.first, let last = trimmed.last, + first == last, first == "\"" || first == "'" else { return nil } + var result = "" + var isEscaped = false + for character in trimmed.dropFirst().dropLast() { + if isEscaped { + result.append(unescape(character)) + isEscaped = false + continue + } + if character == "\\" { + isEscaped = true + continue + } + result.append(character) + } + return result + } + + private static func splitArguments(_ characters: [Character]) throws -> [String] { + var arguments: [String] = [] + var current = "" + var depth = 0 + var index = 0 + + while index < characters.count { + let character = characters[index] + if character == "\"" || character == "'" { + current += try readStringLiteral(characters, from: &index) + continue + } + if character == "(" || character == "[" || character == "{" { depth += 1 } + if character == ")" || character == "]" || character == "}" { depth -= 1 } + if character == ",", depth == 0 { + arguments.append(current) + current = "" + index += 1 + continue + } + current.append(character) + index += 1 + } + + if !unwrapped(current).isEmpty { arguments.append(current) } + return arguments + } + + private static func matchingParenthesis(_ characters: [Character], openIndex: Int) -> Int? { + var depth = 0 + var index = openIndex + while index < characters.count { + let character = characters[index] + if character == "\"" || character == "'" { + guard (try? readStringLiteral(characters, from: &index)) != nil else { return nil } + continue + } + if character == "(" { depth += 1 } + if character == ")" { + depth -= 1 + if depth == 0 { return index } + } + index += 1 + } + return nil + } + + private static func readStringLiteral(_ characters: [Character], from index: inout Int) throws -> String { + let quote = characters[index] + var literal = String(quote) + var cursor = index + 1 + + while cursor < characters.count { + let character = characters[cursor] + literal.append(character) + if character == "\\" { + guard cursor + 1 < characters.count else { throw MongoShellValueError.unterminatedString } + literal.append(characters[cursor + 1]) + cursor += 2 + continue + } + if character == quote { + index = cursor + 1 + return literal + } + cursor += 1 + } + throw MongoShellValueError.unterminatedString + } + + // MARK: - Scalars + + private static func isIdentifierStart(_ character: Character) -> Bool { + character.isLetter || character == "_" || character == "$" + } + + private static func isIdentifierPart(_ character: Character) -> Bool { + isIdentifierStart(character) || character.isNumber + } + + private static func unwrapped(_ value: String) -> String { + value.trimmingCharacters(in: .whitespacesAndNewlines) + } + + private static func unescape(_ character: Character) -> Character { + switch character { + case "n": return "\n" + case "t": return "\t" + case "r": return "\r" + default: return character + } + } + + private static func escaped(_ value: String) -> String { + value.replacingOccurrences(of: "\\", with: "\\\\").replacingOccurrences(of: "\"", with: "\\\"") + } + + private static func hexData(_ value: String) -> Data? { + let characters = Array(value) + guard !characters.isEmpty, characters.count.isMultiple(of: 2) else { return nil } + var bytes: [UInt8] = [] + bytes.reserveCapacity(characters.count / 2) + var index = 0 + while index < characters.count { + guard let byte = UInt8(String(characters[index ... index + 1]), radix: 16) else { return nil } + bytes.append(byte) + index += 2 + } + return Data(bytes) + } +} diff --git a/TablePro/Core/Plugins/PluginMetadataRegistry+RegistryDefaults.swift b/TablePro/Core/Plugins/PluginMetadataRegistry+RegistryDefaults.swift index 74f71b4ca..09f7a3343 100644 --- a/TablePro/Core/Plugins/PluginMetadataRegistry+RegistryDefaults.swift +++ b/TablePro/Core/Plugins/PluginMetadataRegistry+RegistryDefaults.swift @@ -100,9 +100,9 @@ extension PluginMetadataRegistry { label: String(localized: "Legacy UUID Encoding"), fieldType: .dropdown(options: [ .init(value: "", label: String(localized: "Do Not Decode")), - .init(value: "javaLegacy", label: String(localized: "Java")), - .init(value: "csharpLegacy", label: String(localized: "C#")), - .init(value: "pythonLegacy", label: String(localized: "Python")) + .init(value: "javaLegacy", label: "Java"), + .init(value: "csharpLegacy", label: "C#"), + .init(value: "pythonLegacy", label: "Python") ]), section: .advanced ) diff --git a/TablePro/Resources/Localizable.xcstrings b/TablePro/Resources/Localizable.xcstrings index 7692425f5..a3bdb057c 100644 --- a/TablePro/Resources/Localizable.xcstrings +++ b/TablePro/Resources/Localizable.xcstrings @@ -140,6 +140,12 @@ } } } + }, + "Do Not Decode" : { + + }, + "Legacy UUID Encoding" : { + }, "—" : { "extractionState" : "stale", diff --git a/TableProTests/Core/MongoDB/BsonDocumentFlattenerTests.swift b/TableProTests/Core/MongoDB/BsonDocumentFlattenerTests.swift index 64a6a0f76..cda948e92 100644 --- a/TableProTests/Core/MongoDB/BsonDocumentFlattenerTests.swift +++ b/TableProTests/Core/MongoDB/BsonDocumentFlattenerTests.swift @@ -237,8 +237,15 @@ struct BsonDocumentFlattenerTests { @Test("A legacy UUID column keeps the binary type name until a representation is set") func undecodedColumnStaysBinary() { let kinds = FlattenFixture.kinds(Self.legacyDocs(), columns: ["id"]) - #expect(kinds == [.binary]) - #expect(BsonDocumentFlattener.typeName(for: .binary, representation: .unspecified) == "BLOB") + #expect(kinds == [.binary(subtype: 3)]) + #expect( + BsonDocumentFlattener.typeName(for: .binary(subtype: 3), representation: .unspecified) + == "BLOB(3)" + ) + #expect( + BsonDocumentFlattener.typeName(for: .binary(subtype: 0), representation: .unspecified) + == "BLOB" + ) } @Test("A configured legacy UUID column reports the representation in its type name") @@ -285,7 +292,7 @@ struct BsonDocumentFlattenerTests { docs.append(["id": MongoDBBinaryValue(data: Data(BsonUuidFixture.javaBytes), subtype: 0x03)]) let kinds = FlattenFixture.kinds(docs, columns: ["id"], representation: .javaLegacy) - #expect(kinds == [.binary]) + #expect(kinds == [.binary(subtype: 0)]) let rows = FlattenFixture.rows(docs, columns: ["id"], representation: .javaLegacy) #expect(rows.allSatisfy { $0[0].asText == nil }) diff --git a/TableProTests/Core/MongoDB/MongoShellValueTranslatorTests.swift b/TableProTests/Core/MongoDB/MongoShellValueTranslatorTests.swift new file mode 100644 index 000000000..e6808a7d9 --- /dev/null +++ b/TableProTests/Core/MongoDB/MongoShellValueTranslatorTests.swift @@ -0,0 +1,239 @@ +// +// MongoShellValueTranslatorTests.swift +// TableProTests +// + +import Foundation +import TableProPluginKit +import Testing + +@Suite("MongoDB Shell Value Translator") +struct MongoShellValueTranslatorTests { + private static let oid = "507f1f77bcf86cd799439011" + private static let uuid = "8cd003eb-4a25-4324-9332-88fce2da0d1a" + + @Suite("Constructors") + struct ConstructorTests { + @Test("ObjectId becomes $oid") + func objectId() throws { + let result = try MongoShellValueTranslator.translate("{\"_id\": ObjectId(\"507f1f77bcf86cd799439011\")}") + #expect(result == "{\"_id\": {\"$oid\": \"507f1f77bcf86cd799439011\"}}") + } + + @Test("ISODate and Date become $date") + func dates() throws { + let iso = try MongoShellValueTranslator.translate("{at: ISODate(\"2026-01-01T00:00:00Z\")}") + #expect(iso == "{at: {\"$date\": \"2026-01-01T00:00:00Z\"}}") + let date = try MongoShellValueTranslator.translate("{at: Date(\"2026-01-01T00:00:00Z\")}") + #expect(date == "{at: {\"$date\": \"2026-01-01T00:00:00Z\"}}") + } + + @Test("Numeric helpers accept both quoted and bare arguments") + func numbers() throws { + #expect(try MongoShellValueTranslator.translate("NumberInt(42)") == "{\"$numberInt\": \"42\"}") + #expect(try MongoShellValueTranslator.translate("NumberInt(\"42\")") == "{\"$numberInt\": \"42\"}") + #expect( + try MongoShellValueTranslator.translate("NumberLong(9007199254740993)") + == "{\"$numberLong\": \"9007199254740993\"}" + ) + #expect( + try MongoShellValueTranslator.translate("NumberDecimal(\"1.25\")") + == "{\"$numberDecimal\": \"1.25\"}" + ) + } + + @Test("Timestamp becomes $timestamp") + func timestamp() throws { + let result = try MongoShellValueTranslator.translate("{ts: Timestamp(1700000000, 1)}") + #expect(result == "{ts: {\"$timestamp\": {\"t\": 1700000000, \"i\": 1}}}") + } + + @Test("BinData keeps its subtype") + func binData() throws { + let result = try MongoShellValueTranslator.translate("{b: BinData(5, \"3q2+7w==\")}") + #expect(result == "{b: {\"$binary\": {\"base64\": \"3q2+7w==\", \"subType\": \"05\"}}}") + } + + @Test("HexData converts to base64") + func hexData() throws { + let result = try MongoShellValueTranslator.translate("{b: HexData(0, \"deadbeef\")}") + #expect(result == "{b: {\"$binary\": {\"base64\": \"3q2+7w==\", \"subType\": \"00\"}}}") + } + + @Test("MinKey and MaxKey work with and without parentheses") + func extremeKeys() throws { + #expect(try MongoShellValueTranslator.translate("{a: MinKey}") == "{a: {\"$minKey\": 1}}") + #expect(try MongoShellValueTranslator.translate("{a: MinKey()}") == "{a: {\"$minKey\": 1}}") + #expect(try MongoShellValueTranslator.translate("{a: MaxKey}") == "{a: {\"$maxKey\": 1}}") + } + + @Test("The UUID family routes through the shared codec") + func uuidFamily() throws { + let standard = try MongoShellValueTranslator.translate("UUID(\"8cd003eb-4a25-4324-9332-88fce2da0d1a\")") + #expect(standard == "{\"$binary\": {\"base64\": \"jNAD60olQySTMoj84toNGg==\", \"subType\": \"04\"}}") + + let java = try MongoShellValueTranslator + .translate("LegacyJavaUUID(\"8cd003eb-4a25-4324-9332-88fce2da0d1a\")") + #expect(java == "{\"$binary\": {\"base64\": \"JEMlSusD0IwaDdri/Igykw==\", \"subType\": \"03\"}}") + + let alias = try MongoShellValueTranslator.translate("JUUID(\"8cd003eb-4a25-4324-9332-88fce2da0d1a\")") + #expect(alias == java) + } + } + + @Suite("Scanning") + struct ScanningTests { + /// A method call is always preceded by a dot; a constructor never is. + @Test("Method names that look like helpers are left alone") + func methodsAreNotTranslated() throws { + let input = "db.users.find({}).sort({a: 1}).limit(10)" + #expect(try MongoShellValueTranslator.translate(input) == input) + } + + @Test("A helper inside a double-quoted string is not translated") + func doubleQuotedStringsAreInert() throws { + let input = "{name: \"ObjectId(\\\"abc\\\")\"}" + #expect(try MongoShellValueTranslator.translate(input) == input) + } + + @Test("A helper inside a single-quoted string is not translated") + func singleQuotedStringsAreInert() throws { + let input = "{name: 'MinKey and ISODate(1)'}" + #expect(try MongoShellValueTranslator.translate(input) == input) + } + + @Test("A field literally named like a helper is untouched") + func fieldNamesAreInert() throws { + let input = "{\"ObjectId\": 1, \"MinKey\": 2}" + #expect(try MongoShellValueTranslator.translate(input) == input) + } + + @Test("Helpers nested in arrays and subdocuments are translated") + func nestedHelpers() throws { + let result = try MongoShellValueTranslator.translate( + "{$or: [{_id: ObjectId(\"507f1f77bcf86cd799439011\")}, {n: NumberInt(1)}]}" + ) + #expect(result == "{$or: [{_id: {\"$oid\": \"507f1f77bcf86cd799439011\"}}, {n: {\"$numberInt\": \"1\"}}]}") + } + + @Test("Translation is idempotent") + func idempotent() throws { + let once = try MongoShellValueTranslator.translate( + "db.c.find({_id: ObjectId(\"507f1f77bcf86cd799439011\"), at: ISODate(\"2026-01-01T00:00:00Z\")})" + ) + #expect(try MongoShellValueTranslator.translate(once) == once) + } + + @Test("Text with no helper is returned unchanged") + func passthrough() throws { + let input = "db.users.find({age: {$gt: 21}}).limit(5)" + #expect(try MongoShellValueTranslator.translate(input) == input) + } + + @Test("An unknown constructor is left for the driver to reject") + func unknownConstructorUntouched() throws { + let input = "{a: SomethingElse(1)}" + #expect(try MongoShellValueTranslator.translate(input) == input) + } + } + + @Suite("Errors") + struct ErrorTests { + @Test("A malformed ObjectId is rejected by name") + func badObjectId() { + #expect(throws: MongoShellValueError.malformedArguments(helper: "ObjectId")) { + try MongoShellValueTranslator.translate("{_id: ObjectId(\"nope\")}") + } + } + + @Test("A non-numeric NumberInt is rejected") + func badNumber() { + #expect(throws: MongoShellValueError.malformedArguments(helper: "NumberInt")) { + try MongoShellValueTranslator.translate("{n: NumberInt(\"abc\")}") + } + } + + @Test("An out-of-range NumberInt is rejected") + func overflowingNumber() { + #expect(throws: MongoShellValueError.malformedArguments(helper: "NumberInt")) { + try MongoShellValueTranslator.translate("{n: NumberInt(99999999999)}") + } + } + + @Test("A helper missing its closing parenthesis is reported") + func unterminatedCall() { + #expect(throws: MongoShellValueError.unterminatedCall(helper: "ObjectId")) { + try MongoShellValueTranslator.translate("{_id: ObjectId(\"507f1f77bcf86cd799439011\"") + } + } + + @Test("An unterminated string is reported once a helper puts the scanner to work") + func unterminatedString() { + #expect(throws: MongoShellValueError.unterminatedString) { + try MongoShellValueTranslator + .translate("{a: ObjectId(\"507f1f77bcf86cd799439011\"), name: \"oops}") + } + } + + /// The translator is not a JSON validator: text carrying no helper is handed to the + /// driver untouched so libbson reports the syntax error. + @Test("Malformed text with no helper is passed through rather than rejected here") + func malformedTextWithoutHelperPassesThrough() throws { + let input = "{name: \"oops}" + #expect(try MongoShellValueTranslator.translate(input) == input) + } + } + + @Suite("Parser integration") + struct ParserIntegrationTests { + @Test("A find filter carrying ObjectId reaches the driver as Extended JSON") + func findFilter() throws { + let operation = try MongoShellParser.parse( + "db.users.find({_id: ObjectId(\"507f1f77bcf86cd799439011\")})" + ) + guard case .find(let collection, let filter, _) = operation else { + Issue.record("expected a find operation") + return + } + #expect(collection == "users") + #expect(filter.contains("\"$oid\"")) + #expect(!filter.contains("ObjectId(")) + } + + @Test("A UUID pasted from the grid is accepted in a filter") + func uuidFilter() throws { + let operation = try MongoShellParser.parse( + "db.docs.find({ref: LegacyJavaUUID(\"8cd003eb-4a25-4324-9332-88fce2da0d1a\")})" + ) + guard case .find(_, let filter, _) = operation else { + Issue.record("expected a find operation") + return + } + #expect(filter.contains("\"subType\": \"03\"")) + } + + @Test("A helper inside sort is translated too") + func sortOption() throws { + let operation = try MongoShellParser.parse( + "db.docs.find({}).sort({at: NumberInt(-1)})" + ) + guard case .find(_, _, let options) = operation else { + Issue.record("expected a find operation") + return + } + #expect(options.sort?.contains("$numberInt") == true) + } + + @Test("A runCommand payload is translated") + func runCommand() throws { + let operation = try MongoShellParser.parse( + "db.runCommand({find: \"c\", filter: {_id: ObjectId(\"507f1f77bcf86cd799439011\")}})" + ) + guard case .runCommand(let command) = operation else { + Issue.record("expected a runCommand operation") + return + } + #expect(command.contains("\"$oid\"")) + } + } +} diff --git a/TableProTests/Plugins/MQLExportHelpersTests.swift b/TableProTests/Plugins/MQLExportHelpersTests.swift index c976293ce..d8db0b1b0 100644 --- a/TableProTests/Plugins/MQLExportHelpersTests.swift +++ b/TableProTests/Plugins/MQLExportHelpersTests.swift @@ -11,16 +11,37 @@ import Testing struct MQLExportHelpersTests { private static let uuid = "8cd003eb-4a25-4324-9332-88fce2da0d1a" - @Test("A legacy UUID exports as BSON binary so the dump re-imports as binary") - func legacyUuidExportsAsBinary() { + /// The dump is a mongosh script, so a value has to be a constructor call. mongosh reads + /// `{"$binary": ...}` as a plain object literal and would insert a subdocument. + @Test("A legacy UUID exports as a BinData constructor carrying its subtype") + func legacyUuidExportsAsBinData() { let value = MQLExportHelpers.mqlJsonValue(for: "LegacyJavaUUID(\"\(Self.uuid)\")") - #expect(value == "{\"$binary\": {\"base64\": \"JEMlSusD0IwaDdri/Igykw==\", \"subType\": \"03\"}}") + #expect(value == "BinData(3, \"JEMlSusD0IwaDdri/Igykw==\")") } - @Test("A standard UUID exports as BSON binary subtype 4") - func standardUuidExportsAsBinary() { + @Test("A standard UUID exports as a BinData constructor with subtype 4") + func standardUuidExportsAsBinData() { let value = MQLExportHelpers.mqlJsonValue(for: "UUID(\"\(Self.uuid)\")") - #expect(value == "{\"$binary\": {\"base64\": \"jNAD60olQySTMoj84toNGg==\", \"subType\": \"04\"}}") + #expect(value == "BinData(4, \"jNAD60olQySTMoj84toNGg==\")") + } + + @Test("Raw binary keeps the subtype its column reported") + func rawBinaryKeepsSubtype() { + let data = Data([0xDE, 0xAD, 0xBE, 0xEF]) + #expect(MQLExportHelpers.mqlBinaryValue(for: data, subtype: 5) == "BinData(5, \"3q2+7w==\")") + #expect(MQLExportHelpers.mqlBinaryValue(for: data, subtype: 0) == "BinData(0, \"3q2+7w==\")") + } + + @Test("A column type name carries the subtype back out") + func columnTypeNameRoundTrip() { + #expect(MongoDBUuidCodec.columnTypeName(forSubtype: 0) == "BLOB") + #expect(MongoDBUuidCodec.columnTypeName(forSubtype: 5) == "BLOB(5)") + #expect(MongoDBUuidCodec.binarySubtype(fromColumnTypeName: "BLOB") == 0) + #expect(MongoDBUuidCodec.binarySubtype(fromColumnTypeName: "BLOB(5)") == 5) + #expect(MongoDBUuidCodec.binarySubtype(fromColumnTypeName: "BLOB(128)") == 128) + #expect(MongoDBUuidCodec.binarySubtype(fromColumnTypeName: "") == 0) + #expect(MongoDBUuidCodec.binarySubtype(fromColumnTypeName: "VARCHAR(255)") == 0) + #expect(MongoDBUuidCodec.binarySubtype(fromColumnTypeName: "BLOB(999)") == 0) } @Test("Ordinary values are unaffected", arguments: [ diff --git a/docs/databases/mongodb.mdx b/docs/databases/mongodb.mdx index b7c2fd537..45cb83468 100644 --- a/docs/databases/mongodb.mdx +++ b/docs/databases/mongodb.mdx @@ -85,7 +85,9 @@ The setting only changes how TablePro reads the bytes; it never rewrites what is **Explain**: Press `Cmd+Option+E` on an MQL statement. TablePro converts `find`, `aggregate`, `countDocuments`, update, delete, and `findOneAnd*` calls into `db.runCommand({"explain": ..., "verbosity": "executionStats"})`. -**MQL Shell Queries**: Filters and pipelines are parsed as JSON and Extended JSON, not JavaScript. Quote every key, operators included: `{age: {$gte: 18}}` fails with a parse error. mongosh helpers like `ISODate("2025-01-01")` fail too; write `{"$date": "2025-01-01T00:00:00Z"}`. The editor has no comment syntax, so a `//` line becomes part of the statement. +**MQL Shell Queries**: Filters and pipelines are parsed as JSON and Extended JSON, not JavaScript. Quote every key, operators included: `{age: {$gte: 18}}` fails with a parse error. The editor has no comment syntax, so a `//` line becomes part of the statement. + +Value constructors are translated for you, so a value copied out of the grid can be pasted straight into a filter: `ObjectId`, `ISODate`, `Date`, `NumberInt`, `NumberLong`, `NumberDecimal`, `Timestamp`, `BinData`, `HexData`, `MinKey`, `MaxKey`, `UUID`, and the legacy UUID names. `db.users.find({_id: ObjectId("507f1f77bcf86cd799439011")})` works. A constructor inside a string is left alone, and a method name is never mistaken for one. Anything else JavaScript, including `new Date()`, arithmetic and regex literals like `/abc/i`, is still not evaluated. Find with a filter: