Save Weeks of Engineering Time With OneSchema's Pre-built Validation Library
See how our proprietary validation library saves you the enormous cost of building and maintaining validators in-house.
July 31, 2024
At OneSchema, we’ve created the largest proprietary library of data validations available, designed for all industries. With our pre-built validators, even non-technical team members can configure your template(s) without asking an engineer for a single line of code – all while saving your team the enormous cost of building and continuously maintaining validations every time your database schema changes.
At face, implementing validators can seem simple, but many can come with unanticipated complexity – especially if you want to create a delightful experience for your users.
A classic example of this is date validators.
There are over 30 different input types to manage since date formats differ by region – 01/03/2023 could mean January 3, 2023, but it could also mean March 1, 2023. On top of the myriad of ambiguous formats, they also require you to constantly deal with elusive edge cases like leap years and leading zeroes.
OneSchema’s date validators contain a built-in transformation algorithm that helps automatically correct data by scanning your customer’s uploaded data, identifying its current format, and adjusting all values to a consistent output. To provide a seamless user experience, the platform can intelligently infer the original date format in a column, even when it may be ambiguous to a person (e.g. MM/DD/YYYY dates vs DD/MM/YYYY dates).
Properly testing to ensure your validator appropriately handles the endless possible combinations of validation errors takes considerable time from your team that could instead be spent on core product functionality. Missing an edge-case could result in needing to scrap uploaded data and result in frustrating manual work for your customers.
Companies often start with a basic implementation of validators, only to be forced to constantly spend time adding in coverage for complex edge cases on their roadmap after being bombarded by customer feedback.
Heron Data experienced the issue immediately when their first iteration of CSV import lacked validation and guidance for their users. Dominik Kwok, CTO at Heron Data recounted that"[Import] was a top cause of frustration for both our customers and our team. We also knew that only some customers would file support tickets. Others might just give up and fail to get value from our platform."
Because of their several implicit validation requirements, their team would encounter CSV upload issues from their users dozens of times a week. Each instance would require hours of manual debugging from a member of Heron’s team, leading them to seek a better solution. After switching to OneSchema’s importer and built-in validation library, their CSV import support tickets went from dozens a week, to effectively 0.
In addition to building well thought out validations for each data type you need, an additional layer of logic is needed to convert between formats. OneSchema’s platform automatically converts between date, number, country code, phone number formats, and more.
Our robust pre-built validations also offer your users a substantially simpler experience when compared with externally-written code. OneSchema's validations include built-in error resolutions for the most common mistakes, offering one click find and replace for data types like picklist and country.
Undoing imports is considerably more complex than simply deleting previously-loaded data. Consider the example of an HR platform where users upload employee data. Once a file is imported, numerous downstream automations (with huge implications) such as creating users and updating payroll may be triggered. In many cases, cleaning up the results of an import with incorrect data validation can be far reaching and take a massive amount of time from your engineering team to fix.
While the resources required to write basic validations may not be high (depending on the use case), the majority of the cost comes in an area that’s not immediately obvious – maintenance.
For Affinity, two years after they started building CSV import, they prioritized our 4th engineering project to add improvements. Rohan Sahai, Director of Engineering at Affinity recalls that "Because it was so challenging to display all of the specific errors that could break the import flow, customers would get esoteric error messages like ‘something is amiss’ whenever there was a missing comma, encoding issue, or a myriad of business-specific data formatting problems that led to downstream processing issues.”
As discussed above, there are an infinite number of edge cases to validate against across data types. Because every spreadsheet uploaded from customers looks different, teams need to continuously test and add coverage for new validation cases as they’re discovered. A large number of encodings, formats, and file sizes also contribute to QA taking an unexpectedly large amount of resources. For most companies, validation becomes a permanent area that needs constant testing.
Another maintenance area where validations require updates is when changes to your database schema are made. For every new data type added, engineering teams need to create new validations. With a tool like OneSchema where all validations come out of the box, database schema updates become trivial.
While OneSchema’s pre-built validators provide in-depth coverage for the most commonly validated data types, we also offer tools that enable verifying custom business logic. Our validation webhooks and in-memory code-hooks can be run in parallel with our pre-built checks to ensure the uploaded data meets all of your requirements.
For companies that are quickly growing and innovating, spending resources on building and maintaining validators takes away valuable engineering time that could be spent elsewhere.
With OneSchema, setting up a validation is as easy as giving the column a name, selecting a validation from the dropdown, and optionally ticking a few boxes. We’re continuously testing our validators against messy CSV data of all types across our customer base, updating them to handle the never-ending list of new edge cases, and adding new ones so your team can focus on building your core product.
You can learn more about OneSchema’s no-code data validation library here.
{{blog-content-cta}}
At OneSchema, we’ve created the largest proprietary library of data validations available, designed for all industries. With our pre-built validators, even non-technical team members can configure your template(s) without asking an engineer for a single line of code – all while saving your team the enormous cost of building and continuously maintaining validations every time your database schema changes.
At face, implementing validators can seem simple, but many can come with unanticipated complexity – especially if you want to create a delightful experience for your users.
A classic example of this is date validators.
There are over 30 different input types to manage since date formats differ by region – 01/03/2023 could mean January 3, 2023, but it could also mean March 1, 2023. On top of the myriad of ambiguous formats, they also require you to constantly deal with elusive edge cases like leap years and leading zeroes.
OneSchema’s date validators contain a built-in transformation algorithm that helps automatically correct data by scanning your customer’s uploaded data, identifying its current format, and adjusting all values to a consistent output. To provide a seamless user experience, the platform can intelligently infer the original date format in a column, even when it may be ambiguous to a person (e.g. MM/DD/YYYY dates vs DD/MM/YYYY dates).
Properly testing to ensure your validator appropriately handles the endless possible combinations of validation errors takes considerable time from your team that could instead be spent on core product functionality. Missing an edge-case could result in needing to scrap uploaded data and result in frustrating manual work for your customers.
Companies often start with a basic implementation of validators, only to be forced to constantly spend time adding in coverage for complex edge cases on their roadmap after being bombarded by customer feedback.
Heron Data experienced the issue immediately when their first iteration of CSV import lacked validation and guidance for their users. Dominik Kwok, CTO at Heron Data recounted that"[Import] was a top cause of frustration for both our customers and our team. We also knew that only some customers would file support tickets. Others might just give up and fail to get value from our platform."
Because of their several implicit validation requirements, their team would encounter CSV upload issues from their users dozens of times a week. Each instance would require hours of manual debugging from a member of Heron’s team, leading them to seek a better solution. After switching to OneSchema’s importer and built-in validation library, their CSV import support tickets went from dozens a week, to effectively 0.
In addition to building well thought out validations for each data type you need, an additional layer of logic is needed to convert between formats. OneSchema’s platform automatically converts between date, number, country code, phone number formats, and more.
Our robust pre-built validations also offer your users a substantially simpler experience when compared with externally-written code. OneSchema's validations include built-in error resolutions for the most common mistakes, offering one click find and replace for data types like picklist and country.
Undoing imports is considerably more complex than simply deleting previously-loaded data. Consider the example of an HR platform where users upload employee data. Once a file is imported, numerous downstream automations (with huge implications) such as creating users and updating payroll may be triggered. In many cases, cleaning up the results of an import with incorrect data validation can be far reaching and take a massive amount of time from your engineering team to fix.
While the resources required to write basic validations may not be high (depending on the use case), the majority of the cost comes in an area that’s not immediately obvious – maintenance.
For Affinity, two years after they started building CSV import, they prioritized our 4th engineering project to add improvements. Rohan Sahai, Director of Engineering at Affinity recalls that "Because it was so challenging to display all of the specific errors that could break the import flow, customers would get esoteric error messages like ‘something is amiss’ whenever there was a missing comma, encoding issue, or a myriad of business-specific data formatting problems that led to downstream processing issues.”
As discussed above, there are an infinite number of edge cases to validate against across data types. Because every spreadsheet uploaded from customers looks different, teams need to continuously test and add coverage for new validation cases as they’re discovered. A large number of encodings, formats, and file sizes also contribute to QA taking an unexpectedly large amount of resources. For most companies, validation becomes a permanent area that needs constant testing.
Another maintenance area where validations require updates is when changes to your database schema are made. For every new data type added, engineering teams need to create new validations. With a tool like OneSchema where all validations come out of the box, database schema updates become trivial.
While OneSchema’s pre-built validators provide in-depth coverage for the most commonly validated data types, we also offer tools that enable verifying custom business logic. Our validation webhooks and in-memory code-hooks can be run in parallel with our pre-built checks to ensure the uploaded data meets all of your requirements.
For companies that are quickly growing and innovating, spending resources on building and maintaining validators takes away valuable engineering time that could be spent elsewhere.
With OneSchema, setting up a validation is as easy as giving the column a name, selecting a validation from the dropdown, and optionally ticking a few boxes. We’re continuously testing our validators against messy CSV data of all types across our customer base, updating them to handle the never-ending list of new edge cases, and adding new ones so your team can focus on building your core product.
You can learn more about OneSchema’s no-code data validation library here.
{{blog-content-cta}}
