The New Computer Update is the most significant upgrade to Open Interpreter since 0.1.0. Almost every component has been rewritten to support our project's objective—building a standard interface between language models and computers.

1. The Computer API

In 2023, Open Interpreter started building the world's first real-time code execution environment for language models.

Now, we're creating an API for language models to use in that environment, starting with basic I/O controls like display, mouse, and keyboard:

computer.display.view() # Returns a screenshot to vision models

computer.mouse.click("On-screen text") # Locates text on-screen and clicks it
computer.mouse.move(icon="Description of icon") # Locates icon on-screen and hovers over it

computer.clipboard.view() # Returns the contents of the user's clipboard

# Full reference: https://docs.openinterpreter.com/computer-api/reference

We are also launching a free preview of the hosted tools that power computer.mouse at api.openinterpreter.com.

2. OS Mode

You can instruct Open Interpreter to use the Computer API to control your computer graphically:

interpreter --os

Even local vision models running via .llamafile, LM Studio, or Jan.ai are supported.

3. LMC Messages

To support the incoming Language Model Computer architecture, the new messaging format that powers Open Interpreter extends OpenAI's messages format to include additional information, and a new role called computer:

[
  {
    "role": "assistant",
    "type": "code",
    "format": "python",
    "content": "plot = create_plot_from_data('data')\ndisplay_as_image(plot)"
  },
  {
    "role": "computer",
    "type": "image",
    "format": "base64.png",
    "content": "base64"
  },
  {
    "role": "assistant",
    "type": "message",
    "content": "Plot generated successfully."
  }
]

Read more about LMC Messages here.

4. Computer Module

The Computer itself has been separated from Open Interpreter's Core, so you can run it independently:

from interpreter import interpreter

for chunk in interpreter.computer.run("python", "print('Hello World!')"):
    print(chunk)

This runs in the same Python instance that interpreter uses, so you can define functions, variables, or log in to services before the AI starts running code.

This also means that you have control over the computer's languages (like Python, Javascript, and Shell) and can easily append custom languages. We have written a guide that shows you how to do this with E2B.

And more...

In addition to these major updates, 0.2.0 comes with a suite of fixes and enhancement from our growing open-source community:

• Fixes crash UnboundLocalError active_block by @CyanideByte in #818

• Inserts dummy api key if missing by @CyanideByte in #808

• Fix README_JA.md by @tegnike in #810

• return empty dict if config file is empty by @sbendary25 in #811

• Add pyautogui mouse click functions and parameters by @Arrendy in #843

• Fixed the error when using Azure OpenAI API by @wsbao in #840

• Add package mismatches info to system_debug_info by @tegnike in #800

• Update keyboard control functions for better input handling by @Arrendy in #845

• implement mouse position function by @Arrendy in #850

• Fixed another a few bugs in using OpenAI API/Azure OpenAI API/OpenAI compatible custom API by @wsbao in #848

• Added new docs for litellm hosted models by @tyfiero in #858

• Added refreshed docs for terminal arguments and python streaming responses by @tyfiero in #864

• Add os docs by @tyfiero in #868

• Fixed the case where UnicodeDecodeError by @Moonlight-YS in #863

New Contributors

• @tegnike made their first contribution in #810

• @sbendary25 made their first contribution in #811

• @Arrendy made their first contribution in #843

• @wsbao made their first contribution in #840

• @tyfiero made their first contribution in #858

• @Moonlight-YS made their first contribution in #863

Full Changelog: v0.1.17...v0.2.0

Note: This update is not backwards compatible. If you use Open Interpreter in Python, please read our migration guide.

This is only the beginning. Happy 2024. ●