Code to Mockup: How AI Can Visualize Your App's UI Without Running It

Recently, I've been exploring various trending GitHub tools focused on AI and developer productivity. The standard procedure for trying them out is always the same: clone, install, and run. Some are even simpler, runnable with a single Docker command. While many of these tools feature a user interface (UI), their documentation often doesn't reveal much about how the UI actually looks. So, the usual simple answer would be to just clone and run the application, right?

This time, I decided to try something different. Instead of running the code, I wanted to see if AI could reverse-engineer the UI solely by reading the code. The goal was to have the AI "calculate" what the UI for a specific view of the application might look like when it runs. I call this concept "virtual screenshots."

Spoiler Alert: I tried this with Gemini AI, and the results truly blew my mind!

Why Would You Do This?

Admittedly, my initial motivation was pure curiosity. However, upon giving it more thought, I realized there are several exciting applications that could serve various purposes:

• Generating Tutorials: Easily create visual guides without needing to run the actual application.
• Product Variations & Previews: Generate virtual screenshots or landing pages for different use cases of your product, offering a sneak peek. Imagine generating different finance app UIs based on code!
• Creating Animations: Use the generated UI structures to animate user interactions.

I experimented with a few applications, and I'll share one example using a simple Gradio app found on Hugging Face. Here’s what I aimed to achieve:

• Provide the AI with the codebase and ask it to generate a "virtual screenshot" as HTML code, which we could then render in a browser.
• Generate virtual screenshots of parts of the screen.
• Generate simple UI animations.

Why HTML?

One might consider generating images, but HTML (or even SVG) offers significantly more flexibility. You can easily fine-tune details by editing the text directly, rather than regenerating the entire thing. Plus, HTML allows for animation using JavaScript, highlighting specific elements, and much more.

Let’s dive into the process:

I chose this fantastic Hugging Face Space for my experiment:

https://huggingface.co/spaces/mozilla-ai/surf-spot-finder

Step-by-Step Guide

Step 1: Clone the Code

First, you need to get the application's source code.

$ git clone https://huggingface.co/spaces/mozilla-ai/surf-spot-finder

If you're cloning from GitHub, you could try the same approach with Deepwiki. I haven’t tried it yet and would be curious to know if it works in the same way as Gemini, as described below.

Step 2: Combine the Codebase

This step involves consolidating all the application's code into a single file. To achieve this, I use an open-source tool called Repomix, which you can also try online.

This makes it easy to copy-paste the entire codebase into the AI model.

ai rabbit /tmp/surf-spot-finder $ npx repomix . 
....

🔎 Security Check:
──────────────────
✔ No suspicious files detected.

📊 Pack Summary:
────────────────
  Total Files: 26 files
  Total Chars: 51,959 chars
 Total Tokens: 11,783 tokens
       Output: repomix-output.xml
     Security: ✔ No suspicious files detected

🎉 All Done!
Your repository has been successfully packed.

Step 3: Generate Virtual Screenshots

As mentioned, the goal is to generate a virtual screenshot in the form of HTML code, which you can then view in any browser or an online HTML viewer like html.onlineviewer.net.

Now, with the combined code ready, open Gemini AI Studio: https://aistudio.google.com/prompts/new_chat.

Start by asking the AI to list the different views within the app:

Generate a list of views in this app

Then, you can either choose the main view or select one of the specific views the AI identified in your codebase. Enter the following prompt and append the code you created in the previous step:

Create an HTML code that looks like a screenshot of this app's main view when the main window is open. You do not generate an image, but an HTML code resembling the screenshot. Here is the repo:

< CODE HERE >

Note: If your application has different views (e.g., preferences, account page, main dashboard), you can specify the exact view you want the AI to generate the screenshot for.

After a few seconds, you should receive HTML code. Copy and paste it into an online HTML viewer like https://html.onlineviewer.net/.

And here is what I got.