I asked Claude to organize my messy Desktop. Here is what happened

In a previous discussion, I explored Claude’s Computer Use capabilities with browser automation. This time, I decided to take a deeper dive by leveraging Claude’s interaction with the entire desktop environment. The journey was both enlightening and challenging, revealing impressive strengths and notable limitations.

Here are some of my past posts about this incredible technology.

Has Anthropic just wiped out an entire industry?

If you have been following the news, you may have read about a new feature (or should I call it a product) in the Claude API - it is called Computer Use. This technology is not just another nice feature of Claude, but could actually be the beginning of the

AI Rabbit News & TutorialsAiRabbit

Will Claude AI Revolutionize the Digital ...

Discover how Anthropic’s Claude Sonnet 3.5 is reshaping the digital gig economy with its groundbreaking Computer Use feature. Read more!

AI Rabbit News & TutorialsAiRabbit

Introducing Desktop Automation with Claude

Claude’s new Computer Use feature allows it to interact with your computer much like a human would—moving the mouse, clicking buttons, reading screens, and more. My experiment involved a straightforward task: organizing files on my desktop based on their content rather than their file type.

But before we go any further, I want to mention two things:

I admit, this is not really my desktop (I don't actually store files on the desktop :).

In a real world scenario I would rather use a fully automated solution to save time and money. But I love to challenge technology with weird use cases, maybe in the future AI will be a piece of cake, who knows.

Here is an interesting Github Project that does just that:
https://github.com/QiuYannnn/Local-File-Organizer

Having said that, let's jump into it.

The Experiment in Action

For this experiment, I placed sample data from the Github Project above on the desktop.

I know a lot of people who would laugh at this mess of files on their cluttered desktops, but let us not overwhelm Claude for now:)

I began with a simple command:

organize my desktop by file content not file type

Claude responded promptly:

"I'll help reorganize the desktop content by subject/purpose rather than file type. First, let's see what files we have to organize..."

Initial Hiccups and Adaptive Strategies

Claude initially encountered errors with the find command:

find: ‘file’: No such file or directory
find: ‘file’: No such file or directory
...

Undeterred, Claude swiftly adapted:

"Let me try a different approach to see what files we have..."

It switched to using ls -R to list directory contents:

Tool Use: bash
Input: {'command': 'ls -R ~/Desktop/'}
/home/computeruse/Desktop/:
1.xlsx
12222_777.docx
IMG_0967.PNG
c8fed35720671596882b0c6a2ad9586d.gif
go_bruin.pptx
logo.png
...

Proactive Tool Installation

Realizing the need for better file analysis, Claude initiated the installation of necessary utilities:

Tool Use: bash
Input: {'command': 'sudo apt-get update && sudo apt-get install -y file poppler-utils'}

The installation process included:

debconf: delaying package configuration, since apt-utils is not installed
...
Setting up file (1:5.41-3ubuntu0.1) ...

This proactive step ensured Claude had the tools required for comprehensive file analysis.

Detailed File Analysis and Organization

With the necessary utilities in place, Claude began analyzing the contents of various files:

=== XLSX Content ===
Data Analysis/1.xlsx: Microsoft Excel 2

=== CSV Content ===
 Ford F150,,,,,,,,,,,
,,,,,,,,,,
Year,,Millage,Price,Link,Place,Owner,,,
...

Based on this analysis, Claude created meaningful folders such as "Vehicle Research," "Company Assets," "Travel Content," and "Financial," organizing files based on their content rather than their extensions.

Verifying Content: Guessing vs. Actual Analysis

Initially, Claude organized the files too quickly, which made me wonder how it was processing the information. Suspecting that Claude might be guessing the content based on file names, I decided to probe further.

I asked Claude:

"Are you guessing the content based on the file names?"

Claude admitted:

"Yes, for some files, I am inferring the content based on their names and extensions."

To ensure accuracy, I instructed Claude to "act like a human being" by opening the files to verify their contents:

User: Act like a human being

Claude began interacting with the GUI, moving the mouse, double-clicking folders, and opening files:

Tool Use: computer
Input: {'action': 'mouse_move', 'coordinates': [x, y]}
Tool Use: computer
Input: {'action': 'double_click'}

It opened a few files and moved them into new folders it created based on the content. Claude successfully opened Excel and image files to view their contents.

.. and a few minutes later

My desktop looked a lot better afterwards. Nice.

However, to be honest, I couldn't be 100% sure that it did this for all files, especially binary ones like `.pptx'. This highlighted both Claude's ability to mimic human actions and the uncertainty in its execution across all file types.

Here are my key takeaways from this simple experiment using Claude Computer Use on the entire desktop.

Pros of Claude’s Desktop Automation

1. Autonomous Navigation and Problem-Solving

Claude demonstrated the ability to self-navigate through different applications, identify issues, and devise solutions. For instance, when faced with an error using the find command:

find: ‘file’: No such file or directory

Claude promptly adapted by switching to an alternative method:

"Let me try a different approach to see what files we have..."

This adaptability suggests that Claude could handle complex tasks independently in the future.

2. Command-Line Efficiency

One of Claude’s standout features is its inclination to seek out command-line alternatives when possible. Instead of relying solely on GUI interactions, Claude often opts for CLI commands, enhancing efficiency and reducing dependency on graphical interfaces. For example, Claude proactively installed necessary utilities:

sudo apt-get update && sudo apt-get install -y file poppler-utils

This proactive step ensured it had the tools needed for comprehensive file analysis, demonstrating a commendable level of initiative.

Cons of Claude’s Desktop Automation

1. Program Switching Challenges

Transitioning between different programs proved to be a significant hurdle. Claude occasionally struggled when moving from one application to another, particularly when initiating new programs. For example, when attempting to open PCManFM for file browsing:

(DISPLAY=:1 pcmanfm &)

Claude faced delays and encountered issues like:

debconf: delaying package configuration, since apt-utils is not installed

These interruptions highlight the challenges Claude faces in maintaining smooth transitions between multiple applications.

2. Time-Consuming GUI Interactions

When tasks required GUI interactions—such as taking screenshots or navigating through folders—Claude’s performance was noticeably slower. These actions depend on the responsiveness of the applications and the underlying system, introducing delays not present in pure command-line operations. For instance:

Tool Use: computer
Input: {'action': 'mouse_move', 'coordinates': [x, y]}

3. Scalability Concerns with Complex Tasks

As the volume of information grows, it’s unclear if Claude can maintain its efficiency and accuracy. The current experiment is a stepping stone, but real-world applications may require more robust solutions to handle extensive and varied data seamlessly.

Such interactions, while adding a layer of realism, can slow down the overall process.

Would I use Computer Use again for this type of task?

The answer is definitely no. At least not for now. Until this moment (November 2024), Computer Use could be a real help for browser automation, but not for complex desktop application automation.

If you want to use it, however, try to limit the application switching and data processing volume, and prepare the desktop environment beforehand (i.e. install the necessary software) and make the prompt very precise to avoid it being annoying, as you can see from the previous example.
And again, I strongly recommend that you try it out with your own use cases to get a feel for the capabilities and limitations.

Here are some more practical usage tips.

Usage Tips

Despite the challenges, the experiment provided valuable insights into Claude’s potential and limitations. Here are two tips to enhance your experience with Claude:

1. Sharing Folders for Seamless Access

You can share specific folders from your personal desktop with Claude by placing them into the ~/.anthropic directory. This folder is mounted into the container where Claude operates, allowing seamless access and interaction with your designated files.

2. Creating a Lessons Learned File

Encourage Claude to create a "lessons learned" file within the same shared folder. This file enables Claude to learn from previous interactions and experiences, transforming it into a more effective and reliable companion for managing desktop tasks.

Final Thoughts

Claude's Computer Use feature shows remarkable skills in desktop automation, particularly in understanding the goal and navigating itself through a complex environment (in this use case with even more challenges like different programs, UI and switching between applications). As mentioned in previous blog posts, this technology is already a game changer for browser automation, but in my humble opinion not yet (at least not without further customisation as mentioned above) in a complex setup like a Linux desktop. But with technological advances such as reduced latency and inference time, and also device AI, these kinds of tasks will become significantly faster and easier than we can imagine.