ZENETICS SDK Quickstart Guide

Install the ZENETICS SDK

The ZENETICS SDK is provided as Python module an can be installed easily in your environment.

> pip install zenetics
> poetry add zenetics

Setting the Environment with the ZENETICS_API_KEY

To authorize against the ZENETICS API, you need to set the ENV-variable ZENETICS_API_KEY in your environment. Alternatively, you can update your .env file if your project supports it.

export ZENETICS_API_KEY=xxx-xxx-xxx      #replace with your API key

You can access your ZENETICS API Key in the ZENETICS Portal (opens in a new tab). API keys are application specific, and you need to select the application you want to test. Once you have selected the application, you will find your API Key under the menu item Settings in the left sidebar.

The section "API Key - Test Runners" lists all the available API-keys. You can create a new one with simply by clicking Create API Key.

Validating the Setup

You can run the check command of the ZENETICS CLI to validate if your setup is working:

> zenetics check
 
or
 
> poetry run zenetics check

The command will show a success message or list which parts of the setup are missing.

Implementing the Generation Function

The ZENETICS Test Runner, the core component of the SDK, requires access to the results of your application, and optional metadata to run the evaluations for the selected test cases. ZENETICS offers an easy way to connect your application by creating a Python file that implements a single function: def generate(input: string) -> Generation. This function receives an input string and returns an instance of the Generation class with the output and additional information.

See the following code example of the basic structure for the generate-function and the required information to enable ZENETICS to run evaluations on the generated output:

 
from zenetics.models.generation import Generation, TokenUsage, ModelMetadata
 
 
# This is the only function that you need to provide. It connects the
# ZENETICS Test Runner with your applications and captures the relevant
# information for the evaluation process.
def generate(input: str) -> Generation:
 
    # Generate the output using your LLM application.
    #
    # output:            The result generated by your LLM application.
    #                    (type: str)
    #
    # retrieval_context: The optional retrieval context for RAG applications. This
    #                    information is required for Relevance evaluators like
    #                    Answer Relevance, Context Relevance, and Groundedness.
    #                   (type: List[str])
    #
    # Replace `YOUR_APPLICATION.generate` with the function that generates the output
    #
    output, retrieval_context = YOUR_APPLICATION.generate(input)
 
    # You can also provide additional information that helps you to better understand
    # the performance and characteristics of your model. We recommend that you provide
    # both TokenUsage and ModelMetadata information.
    output, retrieval_context, token_usage, metadata = YOUR_APPLICATION.generate(input)
 
 
    # The Generation object contains the necessary information for the
    # ZENETICS Test Runner to evaluate the specific test case and perform the
    # evaluations assigned to to the test suite.
    return Generation(
        output=output,
        retrieval_context=retrieval_context,
 
        # Optional: Token usage helps you track the tokens required for
        # generating results.
        token_usage=TokenUsage(    # replace this with the data returned from your app.
            input_tokens=864,
            completion_tokens=567,
            total_tokens=1431
        ),
        # Optional: Metadata helps you relate test results to your specific
        # model configurations.
        metadata=ModelMetadata(    # replace this with the actual metadata from your app.
            provider="OpenAI",
            model= "gpt-3.5-turbo",
            parameters={
                "max_tokens": 1024,
                "temperature": 0.7
            },
        )
    )

You have different options to integrate your application via the generate-function:

• Direct Call: Import your application into the file and call the function directly.
• API Call: Request the output and information via an API call to your application.

Select Your Test Suite

Now you are ready to run your first test suite via the ZENETICS CLI. The CLI provides a simple interface to run test suites on your local machine or in your CI/CD system.

You will need the ID of the Test suites you want to run. You can get the Id via the CLI or from the ZENETICS Portal.

Run the following command in your CLI to list all test suites of your application:

> zenetics testsuites
> poetry run zenetics testsuites

Alternative, you can get the tet suite id from the list of test suites in your application. Go to the ZENETICS Portal (opens in a new tab) and select the application you want to test. In the left sidebar, you will find the menu item Test Suites. Click on it to see the list of available test suites.

Hover over the Id column of the list to easily copy the Id of your selected test suite.

Run Your Test Suite

Now you are ready to run your first test suite via the ZENETICS CLI. The CLI provides a simple interface to run test suites on your local machine or in your CI/CD system.

You can run a selected test suite with the following example command.

> zenetics run 1234 src/test/generate.py
 
or
 
> poetry zenetics run 1234 src/test/generate.py

The CLI support several parameters to customize the test run:

• --verbose: Prints also passed test cases in the result summary
• --local-only: Only creates a local test run file but no test run in ZENETICS

The console will show a summary of the test run.

Review Test Run Result

Congratulations 🎉, you just ran your first test suite. Now, let's inspect the results in ZENETICS.

You can use the ZENETICS Test Run URL shown at the end of the test results in the CLI or open your ZENETICS Account (opens in a new tab). Once you select the application, you will automatically see the list of Test Runs.

Feel free to dive deeper into the results and explore the detailed evaluations of your test cases.

You successfully completed all the steps to get you started to test your applications with ZENETICS. Feel free to head back to the ZENETICS Quickstart Guide