GEOWEAVER

When it comes to full-stack AI workflows every step matters!

Open/Web Based

Geoweaver is an open source web-based application allowing users to compose, execute, and share full-stack distributed AI workflows.

Python driven/OS of choice

Geoweaver supports Python and shell scripting. Geowever can be installed on Linux, Mac, and Windows.

Share workflows

Full-stack workflows can be exported/imported as a single package file. That package file includes source code and execution history.

Provenance

Geoweaver records the history of each execution and each change to the code, whether in Python, Jupyter notebooks, or a shell script.

example weaver graphic

Weaver: Build your AI workflow!

Geoweaver takes every step of a full-stack AI workflow and translates it into a reproducible, sharable push-button solution.

Geoweaver (version 0.9.7) is currently supported by ESIP, NSF geoinformatics program #1947893 & #1947875, and NASA ACCESS-19.

Geoweaver is a community effort and we welcome all contributors. If you want to get involved, please create your first issue at Github.

Geoweaver Services


Quick Start Guide

You will need to know your Geoweaver password before you start. The default password can be found when Geoweaver is first started. If you don't know the password, please use command "java -jar geoweaver.jar resetpassword" to reset it.

1. Create and run a Process

  • Click the icon after 'Process' on the righthand menu.
  • Select "Python" from the Language dropdown.
  • Input "helloworld" for the name.
  • Add the code print 'hello world' to the code area.
  • Click "Add" on the bottom. A new process "helloworld" will be added to the Process/Python basket.
  • Click on the newly added helloworld process. An informational panel will show in the main area.
  • Click to run the process. In the popup window, select "Localhost" and click "Execute". In the popup Python environment dialog, click "Confirm" to the default. Add your password.
  • If you see hello world printed in the logging window, it means you have successfully created and run your first process in Geoweaver! Congrats!

2. Create and Run a Workflow

  • Click twice on after the "helloworld" Python process created in Step 1. The main work area will switch to the "Weaver" tab and two circles containing "Hello World" should appear in the workspace.
  • Link the two circles by dragging from one circle and dropping on another circle while pressing "SHIFT" on the keyboard.
  • Click on the "Options" toolbar on the left side of the workspace. In the popup window, input "HelloWorldWorkflow" for the workflow name, and type "Testing hello world" in the Description field. Click "Confirm".
  • To run the workflow, click . In the popup window, select "one-host" option. Choose localhost and set the environment to the default. Click "Run".
  • In the password dialog, enter the password for localhost.
  • While the workflow is in execution mode, blue means the process is waiting, yellow means the corresponding process is running, green means the process execution is finished, and red means the process execution is failed for some reason.
  • Double click on the circles to check the real time output. If users need more details, click "Details" button in the popup information window.

3. Export and share a Workflow

  • Click when the workflow is present in the Weaver workspace.
  • In the popup window, there are two options: workflow with process code / workflow with process code and history. The former will only download source code and workflow json. The latter will download not only source code and workflow, but also all the history details of previous execution of the workflow.
  • Click "Confirm". A ZIP file will be automatically downloaded to your machine.
  • To import a workflow, click . In the upload window, select the ZIP file, click "Start".
  • Once the uploading finished, if the workflow file is valid, it will ask "The upload workflow is valid. Do you want to proceed to save it into the database?" Click "OK".
  • The workflow will be automatically loaded into the Weaver workspace and ready for execution and reuse.
  • Congratulations you have learnt how to use Geoweaver!

Supported Hosts

SSH Server

SSH, or Secure Shell, is a remote administration protocol that allows users to control remote servers over the Internet. The service was created as a secure replacement for the unencrypted Telnet and uses cryptographic techniques to ensure that all communication to and from the remote server happens in an encrypted manner. It provides a mechanism for authenticating a remote user, transferring inputs from the client to the host, and relaying the output back to the client.

Geoweaver provides a Host category called SSH Server to manage all the remote servers with SSH enabled. There is a builtin SSH console to login and type command lines to directly run scripts. Geoweaver also provides a file browser and uploader for users to transfer the files from/to remote servers. Geoweaver can be considered as two-in-one alternative for SSH client and FTP client.

Jupyter NoteBook Server

Jupyter Notebook is an open source web application that allows you to create and share documents that contain live code, equations, visualization, and narrative text.

In Geoweaver, you can manage multiple Jupyter Notebook server in one place. It will provide a proxy for you to record your usage history during eidting the notebooks. All the history will be automatically recorded when you click Save button or shortcut inside the Jupyter Notebook. The history will be retrievable in Geoweaver database and people can always go back and verify their experiments using the history button of the Jupyter host in Geoweaver.

Jupyter Hub

JupyterHub is a multi-user version of the Jupyter Notebook designed for companies, classrooms and research labs.

Geoweaver supports JupyterHub in a similar way to the Jupyter Notebook. People can manage multiple JupyterHub servers in Geoweaver. They can record their usage history in each JupyterHub in Geoweaver database which is queriable in future, even the original JupyterHub is gone. With the history, people can clearly understand what happens and reproduce the experiments with minimum efforts in a new environment.

Jupyter Lab

JupyterLab is a web-based interactive development environment for Jupyter notebooks, code, and data. It is more like a combination of Jupyter Notebook and many other useful tools (e.g., file browser, data visualization module, status monitor, etc).

Geoweaver supports Jupyter Lab in a similar to Jupyter Notebook and JupyterHub. People can manage multiple JupyterLab in Geoweaver and Geoweaver provides a proxy to monitor the traffic between users and Jupyter Lab and record the history in its database. The history database will persist even after the computing server the JupyterLab was hosted is gone. Geoweaver will significantly improve the management of distributed JupyterLab and enhance the transparency, reusuability, and reproducibility of all the work done in JupyterLab.

Processes

Geoweaver supports four types of processes to be executed on the SSH hosts enlisted in the Host section: Shell script, Notebooks, Python code, and Builtin processes.

Shell

Shell scripts can be directly created, saved, executed, and monitored in Geoweaver. Users can execute the shell scripts on remote servers or the localhost server which Geoweaver is hosted.

NoteBook

Geoweaver supports to run Jupyter notebooks using the nbconvert command. The notebook and its logout are recorded in the database.

Build In Process

To help people with limited programming skills, we are developing a set of built-in processes which have fixed program code and expose only input parameters to users. These processes make Geoweaver a powerful AI graphical user interface for the diverse user groups to learn and experiment their AI workflows without coding. Most buil-in processes in Geoweaver are developed based on the existing AI python ecosystem like Keras and Scikit-learn. This section is under intensive development right and the first stable version expects users to create a full-stack AI workflow without writing a single line of code.

Python

Python is one of the most popular AI programming langauge and most AI-related packages reside in Python. Geoweaver supports Python coding and scripting on top of multiple servers while reserving and maintaining the code in one database. All the historical runs are recorded and served in Geoweaver to prevent future duplicated attempts and significantly improve the reproducibility and reusability of AI programs.

Workflow

The workflow module allows users to compose a workflow using the existing processes. The workflow graph composer only supports Directed Acyclic Graph (DAG) (no cyclic loops and bidirectional connections) to chain multiple processes into workflows. Geoweaver provides an all-in-one function to execute, monitor, trace, and display the workflows, and check the history of workflow execution. During the execution, the color of the processes indicate the real-time status: Yellow - Running; Red - Failure; Green - Success.

Please select a host on the right panel or a new host!

Please select a process on the right panel or a new process!

Please select a workflow on the right panel or a new workflow!

Logging

Options
Some message..