⚠ This page is served via a proxy. Original site: https://github.com
This service does not collect credentials or authentication data.
Skip to content

Conversation

@cristipufu
Copy link
Member

@cristipufu cristipufu commented Jan 18, 2026

Description

Decouples mocking from evaluation context.

  • Separates the mocking infrastructure from the evaluation system, enabling mocking to be used independently.
  • Mocking can now be configured without requiring a full evaluation item.

Ref: #1117

Development Package

  • Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath==2.5.10.dev1011483997",

  # Any version from PR
  "uipath>=2.5.10.dev1011480000,<2.5.10.dev1011490000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath>=2.5.10.dev1011480000,<2.5.10.dev1011490000",
]

@cristipufu cristipufu requested a review from akshaylive January 18, 2026 06:34
@cristipufu cristipufu self-assigned this Jan 18, 2026
@cristipufu cristipufu added the build:dev Create a dev build from the pr label Jan 18, 2026
@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Jan 18, 2026
"args": args,
"kwargs": kwargs,
},
"agentInfo": { # This is incomplete
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akshaylive why do we need this agentInfo?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prompts were mostly copied over from URT (check L41). It does make sense for mocker to know about the agent's context -- especially the eval inputs. Maybe we need to create a separate class like this?:

class MockItem(BaseModel): # Equivalent to evaluation item
    inputs: Any
    name: str = Field(default="debug", ..)
    mocking_strategy: MockingStrategy

@cristipufu cristipufu force-pushed the fix/decouple_mocking branch 3 times, most recently from a583ab4 to 0f86c8f Compare January 18, 2026 06:43
@cristipufu cristipufu requested a review from mjnovice January 18, 2026 06:43
@cristipufu cristipufu force-pushed the fix/decouple_mocking branch from 0f86c8f to 631956b Compare January 18, 2026 07:33
evaluation_context: ContextVar[EvaluationItem | None] = ContextVar(
"evaluation", default=None
mocking_strategy_context: ContextVar[MockingStrategy | None] = ContextVar(
"mocking_strategy", default=None
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we actually need this, or can we rely on mocker_context?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The reason is because the mocking strategy can be stateful. For example, a mockito strategy could say "when input is any, return "foo" for the first invocation and "bar" for the second invocation". Without keeping the reference to MockingStrategy in the context, this is hard.

Copy link
Collaborator

@akshaylive akshaylive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great overall. We do need to figure out the details regarding agentInfo and evaluation_criterias in a debug context.

self, eval_item: EvaluationItem, runtime: UiPathRuntimeProtocol
) -> EvaluationItem:
"""Use LLM to generate a mock input for an evaluation item."""
expected_output = (
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's strange for input mocker to be using expectation values. The prompts were reused so we didn't think much of this. Do you know what happens in prod during simulations -- is it {}?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Secondly, evaluation_criterias is a map of evaluator_id -> criterias. For URT/"legacy evaluation items", these are repeated during load, so this object will have a lot of repeated values.

My $0.02 is that we should completely get rid of these fields from input simulation but we can do that in a separate PR. @bai-uipath : could you follow up on this with the right POCs?

"args": args,
"kwargs": kwargs,
},
"agentInfo": { # This is incomplete
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prompts were mostly copied over from URT (check L41). It does make sense for mocker to know about the agent's context -- especially the eval inputs. Maybe we need to create a separate class like this?:

class MockItem(BaseModel): # Equivalent to evaluation item
    inputs: Any
    name: str = Field(default="debug", ..)
    mocking_strategy: MockingStrategy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:dev Create a dev build from the pr test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants