Enhanced file reader script to allow conversion to best datatypes pos…

[kaushik-sb]

[FEATURE] Enhance CSVFileReader to Support Robust dtype Conversion with Null Values

Description

This PR enhances the CSV file reading functionality to handle datatype conversion more robustly, particularly in the presence of null values.

Previously, datasets containing null values could lead to inconsistent or suboptimal datatype inference. For example, integer columns with missing values were often coerced to floating-point types (resulting in values like 1.0 instead of 1), and string columns could be interpreted as generic object types with inconsistent null representations (NaN, None, or empty strings).

With this update, when convert_dtypes is enabled in the source configuration, the dataframe leverages pandas' built-in convert_dtypes() method for safer and more accurate type inference.

Changes Made

• Added conditional dtype conversion in CSVFileReader:

  • If convert_dtypes is set to true in the source configuration (YAML), the dataframe now executes (set to false by default if not specified) :

    result = result.convert_dtypes()

Improvements Introduced

1. Nullable Integer Handling

• Prevents automatic coercion of integer columns with nulls into float64
• Uses pandas' nullable integer types (e.g., Int64)
• Eliminates .0 formatting issues (e.g., 1.0 → 1)

2. Improved String Handling

• Converts generic object dtype columns into pandas' dedicated string dtype
• Ensures consistent representation of textual data
• Improves downstream operations like filtering, comparisons, and transformations

3. Standardized Null Representation

• Replaces inconsistent null markers (NaN, None, etc.) with pandas' unified <NA> representation
• Ensures compatibility across all nullable dtypes (integer, boolean, string)
• Reduces edge-case bugs in data processing pipelines

4. Boolean Type Support

• Enables nullable boolean dtype (boolean) instead of fallback to object
• Supports three states: True, False, and <NA>

5. Overall Data Consistency

• Provides more predictable and semantically correct data types
• Aligns with modern pandas best practices for handling missing data
• Improves interoperability with downstream analytics and processing layers

Definition of Done

Before submitting this pull request, please ensure that the following criteria have been met:

• All automated tests have passed successfully.
• All manual tests have passed successfully.
• Code has been reviewed by at least one other team member.
• Code has been properly documented and commented as needed.
• All new and existing code adheres to our project's coding standards.
• All dependencies have been added or removed from the project's README or other documentation as needed.
• Any relevant documentation or help files have been updated to reflect the changes made in this pull request.
• Any necessary database migrations have been run.
• Any relevant UI changes have been reviewed and approved by the UI/UX team.

Test Evidence

1. CSV file processed without convert_dtypes flag in file source (Has .0 issue in numerical columns that have null entries)

test_without_convert.csv

2. CSV file processed with convert_dtypes flag in file source (Has correct data formatting even though columns have null entries)

test_with_convert.csv

3. Test yml file used for generating above csv test output files

test_convert_dtypes.yml

Additional Notes

[Add any additional notes or context for the reviewer or future maintainers of this code.]

Thank you for submitting!