[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Empty nullable fields return as true when queried from RDS Postgres #11871

Open
1 task done
Neuroforge opened this issue Nov 19, 2024 · 2 comments
Open
1 task done
Labels
aws:rds Amazon Relational Database Service status: backlog Triaged but not yet being worked on type: bug Bug report

Comments

@Neuroforge
Copy link
Neuroforge commented Nov 19, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

When i return an object from a table with a nullable column, if that records value is null... the RDS Data API appears to return TRUE.

Expected Behavior

The column should be set to NULL.

How are you starting LocalStack?

With the localstack script

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

DISABLE_CORS_CHECKS=1 DEBUG=1 localstack start

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Add a table such as this to Postgres v11 or v13 on LocalStack.

CREATE TABLE IF NOT EXISTS my_table (
    my_table_id VARCHAR(36) PRIMARY KEY,
    will_return_ok TEXT NOT NULL,
    will_return_true_if_empty: TEXT[],
);

Add data where will_return_true_if_empty=[], will_return_true_if_empty=['text'], will_return_true_if_empty=NULL

Call this query 'SELECT * FROM my_table;'

export const executeRdsStatement = async (
  queryString: string,
  secretArn: string,
  dbClusterArn: string,
): Promise<DatabaseResponse[]> => {
  const command = new ExecuteStatementCommand({
    database: "root" /* required */,
    formatRecordsAs: "JSON",
    includeResultMetadata: true,
    resourceArn: dbClusterArn,
    secretArn: secretArn,
    sql: queryString /* required */,
  });
  const result = await rdsdataservice.send(command);
  if (result.formattedRecords) {
    return JSON.parse(result.formattedRecords) as DatabaseResponse[];
  }
  return [];
};

Observe that the formatted records have TRUE for the null value record.

Environment

- OS: MacOS
- LocalStack: 
  LocalStack version: 3.8.1
  LocalStack Docker image sha: sha256:ede1832a84149228365c9a431c526da8cc004f3a199d6fc20fc4e3f739b9dd06
  LocalStack build date: 2024-07-16
  LocalStack build git hash: c4457f6

Anything else?

Current workaround is the inspect the table schema and map the column value back. This does not occur in my prod/dev deploys in AWS. The database, when queried directly has the correct values.

const getTableSchema = async (
  secretArn: string,
  dbClusterArn: string,
  tableName: string,
) => {
  const schemaQuery = `
    SELECT column_name, data_type, is_nullable
    FROM information_schema.columns 
    WHERE table_name = $1
    ORDER BY ordinal_position;
  `;

  const command = new ExecuteStatementCommand({
    database: "root" /* required */,
    formatRecordsAs: "JSON",
    includeResultMetadata: true,
    parameters: [{ name: "1", value: { stringValue: tableName } }],
    resourceArn: dbClusterArn,
    secretArn: secretArn,
    sql: schemaQuery,
  });

  const response = await rdsdataservice.send(command);
  if (response.formattedRecords) {
    return JSON.parse(response.formattedRecords);
  }
  return [];
};
@Neuroforge Neuroforge added status: triage needed Requires evaluation by maintainers type: bug Bug report labels Nov 19, 2024
@localstack-bot
Copy link
Collaborator

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

@Neuroforge Neuroforge changed the title bug: Empty nullable fields return as true when queried form RDS Postgres bug: Empty nullable fields return as true when queried from RDS Postgres Nov 19, 2024
@Anze1508 Anze1508 added aws:rds Amazon Relational Database Service status: backlog Triaged but not yet being worked on and removed status: triage needed Requires evaluation by maintainers labels Nov 19, 2024
@Neuroforge
Copy link
Author
Neuroforge commented Nov 20, 2024

Work around for the time being.

  1. Guess the table name form the query, if it is a select query.
  2. Get the information schema for the table.
  3. Transform nullable fields that aren't boolean, to null if they are true.
const getTableSchema = async (tableName: string) => {
  console.log("Getting schema");
  const schemaQuery = `SELECT column_name, data_type, is_nullable FROM information_schema.columns WHERE table_name = '${tableName}' ORDER BY ordinal_position;`;

  const command = new ExecuteStatementCommand({
    database: "root" /* required */,
    formatRecordsAs: "JSON",
    includeResultMetadata: true,
    resourceArn: dbArn,
    secretArn: secretArn,
    sql: schemaQuery,
  });

  const response = await rdsdataservice.send(command);
  console.log(response);
  if (response.formattedRecords) {
    return JSON.parse(response.formattedRecords);
  }
  return [];
};

// Function to validate and identify incorrect TRUE values
const validateRecord = (record: any, schema: any[]) => {
  console.log("Validating record:", record);
  console.log("Schema:", schema);
  for (const column of schema) {
    if (record[column.column_name]) {
      const value = record[column.column_name];
      // Check for incorrect TRUE values in non-boolean fields
      if (
        column.is_nullable &&
        column.data_type !== "boolean" &&
        value === true
      ) {
        console.info(
          `Incorrect TRUE value in column ${column.column_name} with data type ${column.data_type}`
        );
        record[column.column_name] = null;
      }
    }
  }

  return record;
};

//Take the last table name that follows the key word FROM.
//Works for reasonably complicated queries, may need modification to suit your case.
const extractLastTableName = (query: string): string | null => {
  const pattern = /\bfrom\s+([a-zA-Z_][a-zA-Z0-9_\.]*)/gi; // Global, case-insensitive regex
  const matches = [...query.matchAll(pattern)];
  return matches.length > 0 ? matches[matches.length - 1][1] : null; // Return the last match or null
};

const isSelectQuery = (query: string): boolean => {
  const pattern = /^\s*select\b/i; // Case-insensitive regex
  return pattern.test(query);
};

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws:rds Amazon Relational Database Service status: backlog Triaged but not yet being worked on type: bug Bug report
Projects
None yet
Development

No branches or pull requests

3 participants