Using AI to generate real LEGO® sets from text input

LEGO
R
ai
Combining generative AI with R and brickr to design LEGO mosaics that can be built in real life.
Author

Ryan

Published

November 27, 2023

Note

This post (and everything on this website) is written as a LEGO AFOL / giant nerd and is not related to my job or employment with the LEGO® Group.

LEGO® play in the age of AI

Artificial intelligence tools have struggled so far to innovate the LEGO play experience. Sure, there are a few projects out there that combine LEGO bricks with AI, but nothing yet that really improves physical play.

  • The BrickIt app uses AI to scan a pile of physical LEGO bricks and then suggests mini-builds you can create with the detected bricks. Super cool project, but from what I can tell, the mini-builds are human designed, limiting the AI to the image detection.

  • This Image2LEGO project is a really neat attempt to render 3D LEGO models from 2D images. It seems much more academic though rather than a usable solution to build something physical and practical.

  • Image generators can make some cool renderings of potential LEGO sets, but after an initial glance, you realize they contain no actual existing LEGO bricks/elements or structure. They might inspire people to build MOCs, but are impossible to be physically built as rendered.

Even trying to leverage existing AI image generators to create a very simple product - a 2D mosaic similar to the LEGO ART or Mosaic Maker products - fails to render anything remotely buildable with actual bricks.

To test this myself, I gave DALL·E and Midjourney the following prompt and got two very different results.

a realistic LEGO mosaic of the face of a blue and green dinosaur


DALL·E: You can see what it’s trying to do here, but ends up looking like a quilt made out of circuit boards.

Midjourney: This one is awesome, but took the word ‘mosaic’ a bit too literally, as if the ancient Romans used LEGO bricks. We end up with one of those impossible builds.

Interestingly, both tools produced green-skinned theropod dinosaurs with blue back spins and yellow eyes.

Instead, I’m looking for a way to use text-based AI prompts to generate LEGO content, complete with instructions, that can be built in the real world.

Meeting AI halfway

Before I joined The LEGO Group, I developed an R package brickr to programmatically build LEGO sets. Its simplest use is to convert images into 2D LEGO mosaics, rendering both the final image, as well as the instructions and LEGO elements needed to build it. I haven’t thought much it much over the last few years, but I realized it could be a great first attempt at combining AI with physical LEGO bricks.

I’ll let the AI to do what it does best - generating novel images from a text prompt - but stop there. Then brickr can take over and convert the AI into LEGO bricks.

Updating brickr

Most of the brickr tools I wrote 5 years ago could be used unchanged. The twist now is to allow a user-supplied text string to generate a mosaic rather than an image file path.

OpenAI API

I chose to use DALL·E for this project - being the lower quality image generator AI tool out there is actually an asset here. LEGO mosaics are essentially very low resolution images where each pixel is a 1x1 LEGO brick. DALL·E is great at making small 2-D images that look like 1980s video game sprites. These work great for mosaics.

Nerd stuff: API, R, and Python

OpenAI has an official API for Python but not R, but brickr is exclusively an R too. To make the two work together, I used reticulate to write functions in R to run the Python to connect to OpenAI. Super easy to do once you have Python installed in your R environment. I used reticulate’s virtual Python environment.

oai_setup <- function(){
  py_run_string('import openai')
  py_run_string('openai.api_type = YOUR_OPENAI_TYPE')
  py_run_string('openai.api_base = YOUR_OPENAI_BASE')
  py_run_string('openai.api_version = YOUR_OPENAI_VERSION')
  
  py_run_string(glue::glue('openai.api_key =  "{Sys.getenv("OPENAI_API_KEY")}"')) #Saved to environmental vars
  
}

oai_setup()

For the API type, base, and version, I used the values supplied from the DALL-E playground when you like ‘show code’ under a rendered image.

End nerd stuff.

Text to Bricks

The user now can describe an image to render (e.g. ‘a happy dog face’) and pass that text to functions that speak with the OpenAI API.

Nerd stuff: new functions for brickr

The user text fills in the blank of a larger query to create images that work well with brickr:

a sprite image of {a happy dog face} with a white background


oai_generate_image <- function(image_prompt){
  
  py_run_string(glue::glue(
    '
  response = openai.Image.create(
    prompt="a sprite image of {image_prompt} with a white background",
    size="256x256",
    n=1
  )
  
  image_url = response["data"][0]["url"]
  
  image_url
  '
  ))
}

And then a function to grab the AI-generated image from a URL and to prepare it for brickr.

generate_brickr_ai <- function(image_prompt){
  oai_generate_image(image_prompt)
  
  this_img = tempfile()
  download.file(py$image_url, this_img, mode="wb")
  
  png::readPNG(this_img)

}

Pro-tip: Be sure to save the output of generate_brickr_ai() each time you run it. There’s no set.seed() option to recreate the same image again. You also don’t want to incur unnecessary costs from running the API.

End nerd stuff.

This prompt can then be combined with all other existing brickr functions.

"a happy dog face" %>% 
  generate_brickr_ai() %>% # new AI function
  image_to_mosaic(img_size = 48) %>% #brickr to turn an image into bricks, 48 studs wide and high
  build_mosaic() #Display it

And we end up with this fully buildable LEGO set!

brickr also provides tools to print out the list of pieces needed to build this and step-by-step instructions.

More Examples

Prompt: A shining disco heart

#buildtogive


Prompt: A house and a tree


Prompt: Sophia from the Golden Girls

This AI has clearly never watched an episode.


Prompt: A portrait of a wizard with a hat and beard


Building an AI-generated LEGO set

Computer images are cool, but I wanted to build an actual LEGO set using bricks I already have. Luckily, I had an unopened World Map in my house with 11,000 1x1 LEGO tiles ready to be mosaic-fied.

brickr already had features to limit the LEGO colors used in a mosaic (the world map only has 10 colors), change the size of the canvas, and limit to just 1x1 bricks instead of all sizes. The only big change I made was to add a new plotting function to render the image as the 1x1 circles instead of rectangular bricks with studs on a black background.

The prompt for the world’s first (maybe?) AI-generated LEGO set: dinosaurs, obviously

face of a blue green and pink dinosaur with big teeth

For this prompt, I opted for a 32x32 mosaic to be able to build it more quickly IRL.

Nerd stuff: running brickr AI with new conditions 
worldmap_colors <- brickr::lego_colors %>% 
  filter(Color %in% c(
    "Brick yellow", "Bright bluish green", "Bright green",
    "Bright orange", "Br. yellowish green", "Earth blue",
    "Flame yellowish orange", "Medium azur", "Vibrant coral",
    "White"
  ))

mosaic1 <- "face of a blue green and pink dinosaur with big teeth" %>% 
  generate_brickr_ai() 

mosaic1_lego <- mosaic1 %>%
  image_to_mosaic(img_size = 32, #Smaller resolution, but easier
                  use_bricks = "1x1",
                  #A color matching technique that works better with a smaller palette
                  method = "cie2000", 
                  color_table = worldmap_colors) 

mosaic1_lego %>% 
  build_mosaic_dots()

End nerd stuff.

And here we have a pretty bada$$ looking AI-generated LEGO dinosaur! I then printed out the list of pieces I needed and used this image as a building guide.

Presenting…

The final build, in actual physical plastic, on my bookshelf!

I’m not sure if this is actually the world’s first AI-generated physical LEGO build, but probably one of the first. 🤷‍♂️

Next steps

This was a simple build, but a big first step in generating a physical LEGO product from a short text prompt using AI.

There are a bunch of improvements or new techniques I’ll be trying soon.

  • Simplest: improve the prompts and setting in the version of OpenAI’s DALL-E to generate images that work better as LEGO mosaics.
  • Train a custom algorithm so it’s designed around LEGO images.
  • Combine AI tools with brickr’s 3D modeling features to use text to generate fully buildable 3D models, like a house or car.

Install brickr, including the new tile mosaic features on GitHub.

Learn more about brickr.

Follow me on Mastodon at @ryantimpe@fosstodon.org or on LinkedIn!