●

Features, Fixes, Developer

Features, Fixes, Developer

Mar 10, 2024

The New Computer Update II

An expanded Computer API, 5x launch speed, and a GUI-understanding local vision model.



The New Computer Update Part II introduces our first local vision model, a suite of native Mac integrations, 5x launch speed, and dozens of other requested features.


Mac Control

To use this feature, run interpreter --os.

The new calendar, contacts, browser, mail, and sms Computer API modules let Open Interpreter control your Mac's native applications.

For example, Open Interpreter can execute the following code to get a contact, then send a text and an email with an attachment.

# Get Jane's number and send a meeting reminder
phone_number = computer.contacts.get_phone_number("Jane")
computer.sms.send(phone_number, "Hey, I'm emailing you the agenda now.")

# Email Jane the agenda with an attachment
computer.mail.send("jane@email.com", "Agenda", "Please see attached for agenda.", ["path/to/agenda.pdf"])


LLM-first Web Browser

To use this feature, run interpreter --os.

The browser lets Open Interpreter quickly browse the web by querying a web-enabled language model.

for event in ["film festival", "concert", "movie"]:
  computer.browser.search(f"upcoming {event} near Pike Place, Seattle")


Point Model

To use this feature, run interpreter --os.

We have developed a model called point which is capable of locating visual controls precisely. It was designed to run locally on consumer hardware.

We leverage existing open-source models to "semantically search the screen" for text and icons. Language models can then call on this composite model to 'point' at text or icons on the screen:

computer.mouse.click(icon="Play button")

While this model is capable of understanding simple interfaces with discrete icons, we intend to explore more general solutions in the next few weeks.


And more:

  • 5X launch speed

  • Experimental Docker support

  • Export conversation to a Jupyter notebook using %jupyter

  • Experimental one-click installers for Mac, Linux, and Windows

  • Profiles preview (a feature similar to custom GPTs)

…

* Added all docs and folders by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/873

* Add safety and docker support documentation by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/882

* Fix cli crash during screenshot generation by @double-thinker in https://github.com/KillianLucas/open-interpreter/pull/884

* Add documentation for local models by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/889

* Fix config edit mac os by @MikeBirdTech in https://github.com/KillianLucas/open-interpreter/pull/927

* Add reference to open-interpreter-termux in README by @MikeBirdTech in https://github.com/KillianLucas/open-interpreter/pull/906

* Fix feedback from input api key by @Notnaton in https://github.com/KillianLucas/open-interpreter/pull/932

* Fix azure openai coding error by @MonsterCoder in https://github.com/KillianLucas/open-interpreter/pull/924

* Fixed %tokens command by @CyanideByte in https://github.com/KillianLucas/open-interpreter/pull/896

* update documentation to include the parameters by @Notnaton in https://github.com/KillianLucas/open-interpreter/pull/908

* Fix: NameError: name 'computer' is not defined by @MikeBirdTech in https://github.com/KillianLucas/open-interpreter/pull/937

* Update ROADMAP.md by @eltociear in https://github.com/KillianLucas/open-interpreter/pull/861

* Fix small typo in README.md by @jameynakama in https://github.com/KillianLucas/open-interpreter/pull/918

* Update README.md by @codesmith-emmy in https://github.com/KillianLucas/open-interpreter/pull/929

* Update local model docs by @MikeBirdTech in https://github.com/KillianLucas/open-interpreter/pull/931

* Profiles~ by @Notnaton in https://github.com/KillianLucas/open-interpreter/pull/933

* Introduced a new documentation structure, added demos, and more by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/948

* create new cli entry point and update poetry script by @MikeBirdTech in https://github.com/KillianLucas/open-interpreter/pull/947

* Added settings by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/949

* Update All Settings page for Open Interpreter Docs by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/967

* Move the entry point to start_terminal_interface by @Notnaton in https://github.com/KillianLucas/open-interpreter/pull/960

* Try to get some error logs from Markdown() by @Notnaton in https://github.com/KillianLucas/open-interpreter/pull/969

* Add missing llm. by @Notnaton in https://github.com/KillianLucas/open-interpreter/pull/975

* Fix the conversation restoration problem in version 0.2.0 New Computer by @Steve235lab in https://github.com/KillianLucas/open-interpreter/pull/977

* Remove getpass add prompt_toolkit by @Notnaton in https://github.com/KillianLucas/open-interpreter/pull/991

* [WIP] Fix broken unit tests. by @Steve235lab in https://github.com/KillianLucas/open-interpreter/pull/995

* chore: update aifs by @shivenmian in https://github.com/KillianLucas/open-interpreter/pull/997

* added string to multiple python file generation for skill library by @akuten1298 in https://github.com/KillianLucas/open-interpreter/pull/996

* add docs to computer by @birbbit in https://github.com/KillianLucas/open-interpreter/pull/1007

* Option to pass paths to docs.search by @birbbit in https://github.com/KillianLucas/open-interpreter/pull/1008

* Load skills without overwriting them by @birbbit in https://github.com/KillianLucas/open-interpreter/pull/1009

* Litellm custom provider by @Notnaton in https://github.com/KillianLucas/open-interpreter/pull/955

* Jupyter export magic command by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/986

* Docs for magic commands updated by @CyanideByte in https://github.com/KillianLucas/open-interpreter/pull/1004

* Add doc strings to all computer API functions for the LLM to use in the skill library by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/1011

* Fix broken link in lm-studio.mdx by @jorgepvenegas in https://github.com/KillianLucas/open-interpreter/pull/1019

* Update README.md by @MikeBirdTech in https://github.com/KillianLucas/open-interpreter/pull/1017

* Boost performance by lazy loading modules by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/1020

* Quick fix to sync the computer api between processes by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/1022

* fix parsing of a function by @birbbit in https://github.com/KillianLucas/open-interpreter/pull/1024

* Make messages accessible to interpreter by @birbbit in https://github.com/KillianLucas/open-interpreter/pull/1025

* Correct punctuation in the document. by @Dragonchu in https://github.com/KillianLucas/open-interpreter/pull/1033

* ROADMAP.md by @matthewrygh in https://github.com/KillianLucas/open-interpreter/pull/1029

* Resolve #1034 Migrate from appdirs to platformdirs by @lombad in https://github.com/KillianLucas/open-interpreter/pull/1038

* Multi-line input support. by @Steve235lab in https://github.com/KillianLucas/open-interpreter/pull/1048

* Fixed local.py not working on Windows. Added GPU acceleration. by @CyanideByte in https://github.com/KillianLucas/open-interpreter/pull/1043

* Rename the app directory to open-intepreter, and migrate all files from older versions by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/1060

* Expand computer api with texting, mail, contacts, and calendar by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/1061

* Fix profile migration bug by @tyfiero in https://github.com/KillianLucas/open-interpreter/pull/1066

* fix some error when start command on macos by @982945902 in https://github.com/KillianLucas/open-interpreter/pull/1062

## New Contributors

* @double-thinker made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/884

* @MonsterCoder made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/924

* @jameynakama made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/918

* @codesmith-emmy made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/929

* @Steve235lab made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/977

* @shivenmian ❤️ made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/997

* @akuten1298 made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/996

* @birbbit made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/1007

* @jorgepvenegas made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/1019

* @Dragonchu made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/1033

* @matthewrygh made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/1029

* @lombad made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/1038

* @982945902 made their first contribution in https://github.com/KillianLucas/open-interpreter/pull/1062

**Full Changelog**: https://github.com/KillianLucas/open-interpreter/compare/v0.2.0...v0.2.1

Subscribe to future changes

Get notified when we release new features.