I asked Claude to Fill Out My Tax Return. Here is What Happened

In a previous discussion, I explored Claude’s Computer Use capabilities with browser automation. In this blog post, I'm giving Claude Computer another try. This time, I want to see if it can fill out a tax return using data I have stored on my hard disk. I'm intentionally not giving it the data in the prompt; instead, I want to see if it can:

The Challenges:

1. Read the form and determine the required fields.
2. Find the relevant information to fill in those fields from the data on disk.
3. Fill out the form accurately.

For this experiment, I've selected a tax return form, specifically the IRS Form 1040:

https://www.irs.gov/pub/irs-pdf/f1040.pdf

And to make things easier for Claude (and because I initially had trouble with PDF editing), I've generated a simplified web-based version of the same form:

Why the web form and not the PDF editor?
I tried using a PDF editor first, but Claude CS didn't manage to fill it out. I suspect this is more of a driver or compatibility issue. So, I switched to a browser-based version for this test.

I've also created some sample tax data and placed it in the /.antropic folder, which is shared between Claude Computer and my host machine (a Mac). Here's the script I used to generate the data:

#!/bin/bash

YEAR=2023
OUTPUT_DIR="tax_source_${YEAR}"
mkdir -p "$OUTPUT_DIR"

# Generate W-2
cat > "${OUTPUT_DIR}/W2_${YEAR}.txt" << EOF
TECH CORP INC
EIN: 94-1234567
742 CORPORATE WAY
SAN FRANCISCO, CA 94101

Employee: James Wilson
SSN: 453-89-1234
742 Evergreen Terrace
Springfield, CA 94101

Box 1: Wages, tips, other comp - $95,000
Box 2: Federal tax withheld - $18,500
Box 3: Social Security wages - $95,000
Box 4: Social Security tax withheld - $5,890
Box 5: Medicare wages and tips - $95,000
Box 6: Medicare tax withheld - $1,377.50
EOF

# Generate 1099-INT
cat > "${OUTPUT_DIR}/1099INT_${YEAR}.txt" << EOF
MEGA BANK
EIN: 95-9876543
100 FINANCE BLVD
NEW YORK, NY 10001

RECIPIENT: James Wilson
TIN: 453-89-1234

Box 1: Interest income - $1,200
Box 4: Federal tax withheld - $336
EOF

# Generate 1099-DIV
cat > "${OUTPUT_DIR}/1099DIV_${YEAR}.txt" << EOF
INVESTMENT FIRM LLC
EIN: 93-4567890
200 WALL STREET
NEW YORK, NY 10002

RECIPIENT: James Wilson
TIN: 453-89-1234

Box 1a: Total ordinary dividends - $3,500
Box 1b: Qualified dividends - $3,000
Box 4: Federal tax withheld - $980
EOF

# Generate 1098 Mortgage Interest
cat > "${OUTPUT_DIR}/1098_${YEAR}.txt" << EOF
HOMELOANS BANK
EIN: 91-2345678
300 MORTGAGE ROW
CHICAGO, IL 60601

RECIPIENT: James Wilson
TIN: 453-89-1234

Box 1: Mortgage interest received - $12,000
Box 10: Property taxes - $4,200
EOF

# Generate Property Tax Statement
cat > "${OUTPUT_DIR}/PropertyTax_${YEAR}.txt" << EOF
SPRINGFIELD COUNTY TAX COLLECTOR
742 Evergreen Terrace
Springfield, CA 94101

Owner: James Wilson
Parcel: 12-345-678

Annual Tax Amount: $4,200
Paid Date 1: 11/01/${YEAR} - $2,100
Paid Date 2: 02/01/$((YEAR+1)) - $2,100
EOF

# Generate Charitable Contribution Receipt
cat > "${OUTPUT_DIR}/Charity_${YEAR}.txt" << EOF
SPRINGFIELD FOUNDATION
EIN: 92-3456789
800 CHARITY LANE
SPRINGFIELD, CA 94101

Dear James Wilson,

Thank you for your generous contributions in ${YEAR}:
03/15/${YEAR} - $1,000
07/20/${YEAR} - $800
12/15/${YEAR} - $700

Total contributions: $2,500
No goods or services were provided in exchange for these contributions.
EOF

echo "Generated tax source documents in $OUTPUT_DIR"

Having all the data ready, let's go!

The Initial Prompt:

Here's the initial prompt I gave to Claude Computer: "fill in the form in firefox using the fields in /.home/computeruse/.anthropic"

Initially, it started a bit off-track, asking for information that I had already provided in the prompt. So, I simply told Claude to re-read the initial prompt.

Claude Gets to Work:

And it did! It figured out what it needed to do. First, it checked all the files in the specified folder:

Then, it took some screenshots of the Firefox form...

...and started filling in the data, one field at a time:

After a few minutes, it was finished. The form was approximately two pages long.

The Results:

First, Claude summarized all the data it had filled out:

And here's a look at the filled-out form data:

If you look carefully, you'll notice that the zip code in the form doesn't match the data it extracted. Also, the address field is missing. The rest seems more or less fine.

Then came the numbers for the tax return. Looking at the input data for the W-2 salary, I realized I had made an error in the data generation. It should have been 5,000, not 75,000:

Box 1: Wages, tips, other comp - 5,000
Box 2: Federal tax withheld - 8,500
Box 3: Social Security wages - 5,000
Box 4: Social Security tax withheld - 890
Box 5: Medicare wages and tips - 5,000
Box 6: Medicare tax withheld - 377.50

I questioned where the 95,000 figure was coming from to see if Claude could identify and explain its mistake:

Where is the 95.000 coming from?

And remarkably, it figured out the mistake on its own and started correcting it!

Conclusion:

In this experiment, we successfully filled out a simplified tax return using data distributed across multiple files. Claude managed to read the fields and collect the relevant figures, mostly on its own. While most of the data was correct, the form filling was still not perfect. However, it produced a good draft that could potentially save some effort.

Key Observations:

• Browser vs. External Apps: I experienced a lot of hiccups when not working directly in the browser. Claude took too long and sometimes couldn't even manage to click a single field in an external application.
• Data Separation: It's beneficial to separate the data collection process from the form-filling process. As demonstrated, when asked to extract specific data from a collection of files without the context of filling a form, Claude was quite effective at choosing the correct figures. Using Claude (even without computer use) to identify and verify the data first, then instructing CS to fill the form, might be a more reliable workflow.
• Spacing Matters: Fields with little spacing were poorly recognized, leading to incorrect input values (for example, the social security number).

Scoring Claude CS:

Based on the tasks defined at the beginning, I'd give Claude CS the following scores:

1. Read the form: 8/10
2. Collect the relevant data: 8/10
3. Fill in the form: 6/10

I believe that when the data is collected properly from different sources, confirmed by the user (and Claude is quite good at this even without the "computer use" feature), and the forms are browser-based with sufficient spacing, Claude can perform quite well. However, all these aspects still feel like works in progress.

Lastly, as with any AI project, the prompt is essential. For scenarios like this, I'd always recommend a human-in-the-loop approach, explicitly requesting Claude not to guess if it lacks information and instead ask the user for clarification.

At the end of the day, at least for very critical tasks, Claude Computer will be an assistant, not (yet) a replacement, for some time to come.