Generating images with LocalAI using a GPU

• Use model specific trigger words to generate images using the specific training sets used. These are model specific and can be found in the model documentation.

Examples: pixel_art, line_art, comic, anime

• Use negative prompts. Most models support these and you can add them after the positive prompt, separated by “|”. They will cause the model to try to avoid what is described in them.

Examples: mutations, out of frame, blurry, missing legs, missing arms

You can find many more good examples and instructions for optimizing prompts on many sites dedicated to AI, check them out for more inspirations and ideas. Here some short prompts making use of the suggestions above:

close-up: Elegant Woman with golden crown, sweet expression, green eyes, black hair. background: rose garden, style: anime | crossed eyes, hands

bird’s eye view: magnificent victorian city in a light mist, beside the sea, sunset; style: photorealistic | birds, bend walls

mystical: straight forest stone trail, greenery, dense trees, dark, soft fog, night | bright, light

Beside using the image model that comes with the LocalAI All-in-One container images, there are a lot more Text-to-Image models available online, and you can add many of those to you LocalAI instance.

Please note, that these models come with licenses that need to be taken into account. For example, some models do not allow their use with a service that provides an API, like LocalAI does. Please make sure to first check if you are allowed to use a model, before installing it with LocalAI. Also be aware, that these models allow to create copyrighted content, that can cause legal consequences when published online.

Adding new models can’t be done in the WebUI, as there are not many configurations for those available in the LocalAI Gallery. You need to create your own configuration files for that, how this is done is shown with the following examples.

Add Models from Huggingface

huggingface is a site similar to github, that hosts all kind of generative models, datasets and other AI learning related resources from users to be downloaded or included directly in code. To add for example the image model OpenJourney:

• Go to the hugginface model page
• From the filters on the left, select Computer Vision: Text-to-Image
• In the Filter by Name field on top, enter a search term, e.g. openjourney
• Click on the model you would like to use
• Check on its Files and Versions tab, if it has a model_index.json file in the top directory
• Now copy the hugginface uri (prompthero/openjourney) on top of the model page and create models/openjourney.yaml pointing to it like this
  (All model relevant changes to the defaults are in red):

name: OpenJourney 
parameters: 
 model: prompthero/openjourney
backend: diffusers 
step: 25 
f16: true 

diffusers: 
 pipeline_type: StableDiffusionPipeline 
 cuda: true 
 enable_parameters: "negative_prompt,num_inference_steps" 
 scheduler_type: "k_dpmpp_2m"  

usage: | 
       curl http://localhost:8080/v1/images/generations  
         -H "Content-Type: application/json"  
         -d '{ 
           "prompt": "<positive prompt>|<negative prompt>", 
           "step": 25, 
           "size": "512x512",
           "model": "OpenJourney"
         }'

• Restart the LocalAI containers
• Go to Image Creation in the WebUI, and choose the now available OpenJourney from the pull down menu, and give it a prompt

close-up: chameleon sitting in a tree, sunny, photorealism, forest background | mutation, extra legs, disfigured

• When doing this for the first time, LocalAI will trigger the download of the model files, however as there is no output of its progress, you can watch the models/models–prompthero–openjourney folder size, that holds all related files.
• When all files have been download, the generated image is displayed.

This way you can add many Text-to-Image models from hugginface. Please note, that some models don’t provide a fp16 version, and will display an error message indicating this. In these cases the config entry

  f16: false

needs to be set in the yaml file. For a full overview of possible config options and their meaning, please consult the LocalAI yaml documentation for all available options and the LocalAI Image Generation documentation on schedulers and backend parameters.

Add Models from Other Sources

Beside hugginface, there are a lot more models available on other sites, offered as single safetensor files. Civitai for example is a website that hosts primarily image generation models, but it is not possible to directly access them from LocalAI by providing a URI like for hugginface. The model files need to be downloaded manually and then referenced in a config file. To find and add models from Civitai:

• Go to Civitai and click on models
• Here search for the kind of model you are interested in, e.g. “pixel”
• In the results filter for
  Model Type: Checkpoint
• Now choose one of the available models, e.g. dreamshaper-pixelart
• Download the dreamshaperPixelart_v10.safetensors file from the model page and copy it to the models directory
• Now create a dreamshaperPixelart_v10.yaml config file like in the first example, but point directly to the downloaded model name:

name: dreamshaperPixelart_v10 
parameters:  
model: dreamshaperPixelart_v10.safetensors  
backend: diffusers 
step: 25 
f16: true 

diffusers: 
pipeline_type: StableDiffusionPipeline 
cuda: true 
enable_parameters: "negative_prompt,num_inference_steps" 
scheduler_type: "k_dpmpp_2m" 

usage: | 
      curl http://localhost:8080/v1/images/generations  
        -H "Content-Type: application/json"  
        -d '{ 
          "prompt": "<positive prompt>|<negative prompt>", 
          "step": 25, 
          "size": "512x512", 
          "model": "dreamshaperPixelart_v10" 
        }'

• Restart the LocalAI container
• Go to Image Creation in the WebUI, and choose dreamshaperPixelart_v10 from the pull down menu, and give it a prompt.
• Make sure to use the trigger words listed for the model on its model card on civitai. These words trigger the specific training sets and thus style or content used to train the model with. For the just added model this it either pixel_art or pixel_art_style:

pixel-art, close-up: chameleon in a tree, sunny, forest background | mutation, extra legs, disfigured

Using XL models

Up to now models trained with an image size of 512×512 were added, that only generated images of that resolution. To generate larger images, either use an upscale tool, or install XL models, that got trained on images with the size 1024×1024. However, in many cases, these models will not generate proper output when requesting images in lower resolution and you’ll get something like this:

If you get similar output for a model, you can’t use it through the WebUI, as here all request are made for a fixed size of 512×512. To generate images with different resolutions, you will have to directly contact the LocalAI API with e.g. a curl request. One of the more popular models you can use, and base for many others is Stable-Diffusion-XL-1.0, also often abbreviated as SDXL1.0 or simply SDXL:

• Create models/Stable-Diffusion-XL-1.0.yaml and paste the following config

name: Stable-Diffusion-XL-1.0
parameters: 
 model: stabilityai/stable-diffusion-xl-base-1.0  
backend: diffusers 
f16: true 
cuda: true 
threads: 7 
step: 25  
debug: true 
gpu_layers: 10 
diffusers: 
 scheduler_type: k_dpm_2_a  
 cuda: true 

usage: | 
       curl http://localhost:8080/v1/images/generations  
         -H "Content-Type: application/json"  
         -d '{ 
           "prompt": "<positive prompt>|<negative prompt>", 
           "model": "Stable-Diffusion-XL-1.0", 
           "step": 25, 
           "size": "1024x1024" 
         }

• Create a shell script txt2img_sdxl.sh with the curl request:

#!/bin/sh 

positive_prompt="${1:-"pixel-art, close-up: chameleon in a tree, sunny, forest background | mutation, extra legs, disfigured"}" 
negative_prompt="${2:-"mutation, extra legs, disfigured"}" 

curl http://localhost:8080/v1/images/generations  
         -H "Content-Type: application/json"  
         -d '{ 
           "prompt": "'"$positive_prompt"'|'"$negative_prompt"'", 
           "model": "Stable-Diffusion-XL-1.0", 
           "step": 25, 
           "size": "1024x1024" 
         }'

• Now you can run the script, and pass the positive and negative prompt as parameters

./txt2img_sdxl.sh 
"close-up: cameleon; creative, modern, digital, color explosion, vibrant shades; style: expressionism" 
"mutation, disfigured, letters, numbers"

• LocalAI returns a url to the generated image

{“created”:1727819277,”id”:”d1a755c7-0dcc-45f6-84e4-23a79c41456e”,”data”:[{“embedding”:null,”index”:0,”url”:”http://127.0.0.1:8080/generated-images/b643021618459.png”}],”usage”:{“prompt_tokens”:0,”completion_tokens”:0,”total_tokens”:0}}

As you have seen, adding and using image generation models with LocalAI is easy and fun. Since AI image generation has become a huge field, only a small portion of what is possible can be covered in this post. Further functionality like Image-to-Image or Depth-to-Image generation as well as the usage of low-rank adaption (Lora) models or the combination of several models to form AGI (artificial general intelligence) functionality will be covered in coming posts. Happy Hacking!

(Visited 1 times, 1 visits today)