Key value pairs? Forms? #216
Replies: 5 comments 17 replies
-
|
This is a great idea, I think that Docling could definitely tackle this issue by creating a specific pipeline targeting government forms as you mentioned. The technology already exists, I believe it's simply a matter of tailoring the output. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah. It's not even necessarily government forms, but forms generally. So more like key value pair extraction. I think the important part is differentiating between tables and KV pairs, and then figuring out what the output format would be. Should this be an issue? It'd be a major help for some stuff I'm working on so happy to test some things out. |
Beta Was this translation helpful? Give feedback.
-
|
@VickySekhon @ericfeunekes Yes, we are actively working on a form model for exactly this reason. Please stay tuned! If you want to collaborate with us (eg share some of the forms you are interested in ), please let us know. We would be happy to do so. |
Beta Was this translation helpful? Give feedback.
-
|
I have thousands of different forms that I'm looking to run this on. How do you want to connect. |
Beta Was this translation helpful? Give feedback.
-
|
contact us here: deepsearch-core@zurich.ibm.com |
Beta Was this translation helpful? Give feedback.
-
|
Here are some forms you can try from the Canadian tax authority. You can get French and English by clicking the language in the top right https://www.canada.ca/en/revenue-agency/services/forms-publications/forms.html |
Beta Was this translation helpful? Give feedback.
-
|
@ericfeunekes Actually, I am downloading all the forms you listed above and trying to do some regression tests now. I have also my team-members working on propagating the key-value and form regions. So, not forgotten at all and working behind the scenes to get a good first version! |
Beta Was this translation helpful? Give feedback.
-
|
Greetings @PeterStaar-IBM and Team! We are very excited about Docling, but our use case will also be working with the US government IRS Forms, and aiming to extract text in a consistent and reliable manner. I created a few samples, after downloading from the IRS website, and entering fake names and only a fake social security number. In one case, docling is able to extract the names and fake social, but they ended up concatenated together. In the other case, which tests OCR, docling extracts the names, but not the fake social security numbers. I'm happy to share all relates files, but hoping that I'm doing something in error. |
Beta Was this translation helpful? Give feedback.
-
|
@mcalello I actually downloaded all tax-forms and we are now actively working on accurate form understanding. It is however very tedious. Would love for us to collaborate if possible! |
Beta Was this translation helpful? Give feedback.
-
|
How can I help? For this month, I'm mainly focused on the most prevalent forms - starting with IRS Form 1040. Does docling need to be trained on this form to properly extract values? |
Beta Was this translation helpful? Give feedback.
-
|
@mcalello the best help is to gather and collect as diverse (tax-)forms as possible. I you want, we can do this together, just reach out to me! |
Beta Was this translation helpful? Give feedback.
-
|
Dear @PeterStaar-IBM and team. I am also very interested in this conversation, although from a very different sector and use-case. Many thanks for your good work and also for sharing |
Beta Was this translation helpful? Give feedback.
-
|
@Dr-Zorg We are prepping for large scale annotation of forms, primarily for taxes. If you have forms you can share, please let me know. |
Beta Was this translation helpful? Give feedback.
-
|
Dear @PeterStaar-IBM, thanks for your prompt reply. |
Beta Was this translation helpful? Give feedback.
-
|
yes, please write an email to deepsearch-core@zurich.ibm.com. We will follow up! |
Beta Was this translation helpful? Give feedback.
-
|
Hey @PeterStaar-IBM I know you guys are working hard on form data and key value pairs. Has there been any update yet ? |
Beta Was this translation helpful? Give feedback.
-
|
@hisan-ideamaker The data campaign is currently running ;) |
Beta Was this translation helpful? Give feedback.
-
|
I'm also looking for libraries or solutions that can extract key-value pairs and tables from generic forms. Any recommendations or suggestions would be greatly appreciated. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I'm curious if you have thoughts on how to use docling for things like completed government forms? E.g extract all key value pairs from a page.
It seems like a difficult problem because a table recognition model could easily get confused between tables and KV pairs. But I'm bringing it up because the extensibility of this library seems like it offers a great opportunity to build something like this that actually works, particularly with the JSON output.
I'm not aware of any specific model that can do it just yet, but even something like a moderately powerful VLM could be inserted in the pipeline somewhere to predict the KV pairs elements.
So have you thought about how to integrate this? How would you build it into the pipeline somewhere as a prediction, even using a powerful closed source model as a proof of concept initially?
Beta Was this translation helpful? Give feedback.
All reactions