Remote Evaluation

Stand Alone Agent

The agent can run in stand alone mode, by using --standalone option. The SimulEval will kickoff a server that host the agent. For instance, with the agent in Quick Start,

> simuleval --standalone --remote-port 8888 --agent dummy_waitk_text_agent_v1.py
2022-12-06 19:12:26 | INFO | simuleval.cli | Evaluate system: DummyWaitkTextAgent
2022-12-06 19:12:26 | INFO | simuleval.agent_server | Simultaneous Translation Server Started (process id 53902). Listening to port 8888

For detailed RESTful APIs, please see (TODO)

Docker

You can also use a docker image to run the simuleval. An minimal example of Dockerfile is

FROM python:3.8
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install -y \
&& apt-get -y install apt-utils gcc libpq-dev libsndfile-dev
RUN git clone https://github.com/facebookresearch/SimulEval.git
WORKDIR SimulEval
RUN git checkout v1.1.0
RUN pip install -e .
CMD ["simuleval", "--standalone", "--remote-port", "8888", "--agent", "examples/quick_start/dummy_waitk_text_agent_v1.py"]

Build and run the docker image:

cd examples/quick_start && docker build -t simuleval_agent .
docker run -p 8888:8888 simuleval_agent:latest

Remote Evaluation

If there is an agent server or docker image available, (let’s say the one we just kickoff at localhost:8888) We can start a remote evaluator as follow. For simplicity we assume they are on the same machine

simuleval --remote-eval --remote-port 8888 \
    --source source.txt --target target.txt \
    --source-type text --target-type text