Package cloud.filibuster.junit

Annotation Type TestWithFilibuster

  • @Target({ANNOTATION_TYPE,METHOD})
@Retention(RUNTIME)
@Documented
@API(status=STABLE,
     since="5.0")
@ExtendWith(FilibusterTestExtension.class)
@TestTemplate
public @interface TestWithFilibuster
    Filibuster test annotation for JUnit 5.

    This annotation can be used to automatically enable a JUnit5 test, annotated with a Test annotation for use with Filibuster.

    Performs the following lifecycle operations:

    • Start the Filibuster server process (using the Filibuster Python server.)
    • Communicate with the server to provide conditional assertions and test generation automatically with JUnit.
    • Terminate the Filibuster server after execution of the test.

    • Field Detail

      • DISPLAY_NAME_PLACEHOLDER

        static final java.lang.String DISPLAY_NAME_PLACEHOLDER
        Placeholder variable for the test name.

      • CURRENT_ITERATION_PLACEHOLDER

        static final java.lang.String CURRENT_ITERATION_PLACEHOLDER
        Placeholder variable for current iteration.

      • TOTAL_ITERATIONS_PLACEHOLDER

        static final java.lang.String TOTAL_ITERATIONS_PLACEHOLDER
        Placeholder variable for total (max) iterations.

      • SHORT_DISPLAY_NAME

        static final java.lang.String SHORT_DISPLAY_NAME
        Placeholder name used for generated tests.

      • INITIAL_SHORT_DISPLAY_NAME

        static final java.lang.String INITIAL_SHORT_DISPLAY_NAME
        Placeholder name used for the initial generated test.

    • Element Detail

      • initialName

        java.lang.String initialName
        Name of the generated test.
        Returns:
        name of the generated test.
        Default:
        "Fault-free Execution (Filibuster synthesized fault-injection test #{currentIteration})"

      • name

        java.lang.String name
        Name of the generated test.
        Returns:
        name of the generated test.
        Default:
        "Filibuster synthesized fault-injection test #{currentIteration}"

      • maxIterations

        int maxIterations
        Maximum number of tests that will be run by Filibuster.

        JUnit5 doesn't allow for the dynamic generation of tests as tests are executed: the stream of test executions must be materialized before execution starts. Therefore, set an upper bound on the number of tests that will be run.

        Details:

        • Test execution 1 always runs: this is the fault-free execution.
        • Test execution n conditionally runs, if necessary, based on communication with Filibuster server. It will conditionally invoke the beforeEach/afterEach *only* if the test is necessary.
        • Test execution (maxIteration + 1) runs to perform finalization of the test process, but bypasses beforeEach/afterEach, and test body. Therefore, it appears to run, is shown as a test in IntelliJ, but doesn't execute the test.

        Caveats:

        • If maxIteration is not set large enough, Filibuster will fail to explore the entire failure space.
        Returns:
        upper bound on possible Filibuster tests that will be run.
        Default:
        99

      • dynamicReduction

        boolean dynamicReduction
        Should this configuration use dynamic reduction?

        Dynamic reduction incurs overhead and is useless unless testing graphs where depth greater than 2 will be explored. This probably is not the case for most functional tests that use mocks, stubs, or similar and is only useful when standing up deep graphs of multiple services.

        Returns:
        whether dynamic reduction should be used.
        Default:
        false

      • suppressCombinations

        boolean suppressCombinations
        Should this configuration avoid exploring combinations of faults?
        Returns:
        whether combinations of faults will be suppressed from test.
        Default:
        false

      • dataNondeterminism

        boolean dataNondeterminism
        Does this test configuration contain data nondeterminism?

        Specifically, this only refers to whether the RPC messages can be affected across executions from data nondeterminism.

        Use of this option prevents optimizations for dealing with scheduling nondeterminism, as accounting for data nondeterminism effectively prevents dealing with scheduling nondeterminism by omitting invocation details specific to a particular invocation of an RPC.

        For reference:

        • Data nondeterminism: use of timestamps, unique identifiers that differ each time the test is run.
        • Scheduling nondeterminism: loops that issue multiple RPCs to the same RPC method on the same service that differ only by message contents.
        Returns:
        whether data nondeterminism is present in the test.
        Default:
        false

      • avoidRedundantInjections

        boolean avoidRedundantInjections
        Should we avoid redundant fault injections?

        If so, repeated execution of the same RPC method with the same RPC arguments will be skipped for fault injection.

        Option will still show RPCs as redundant in the UX, to avoid set the system property instead.

        Returns:
        whether we should avoid redundant injections
        Default:
        false

      • avoidInjectionsOnOrganicFailures

        boolean avoidInjectionsOnOrganicFailures
        Should we avoid fault injections where RPCs have failed without fault injection?
        Returns:
        whether we should avoid these injections
        Default:
        false

      • failOnOrganicFailures

        boolean failOnOrganicFailures
        Should we fail the test when RPCs have failed without fault injection?
        Returns:
        whether we should avoid these injections
        Default:
        false

      • analysisFile

        java.lang.String analysisFile
        Analysis file that should be used for this configuration of Filibuster.

        When supplied, analysisConfigurationFile() is ignored.

        When not specified, a default analysis file will be generated to test common faults for Google's gRPC. (i.e., DEADLINE_EXCEEDED, UNAVAILABLE)

        Returns:
        absolute path of the analysis file to use.
        Default:
        ""

      • analysisResourceFile

        java.lang.String analysisResourceFile
        Analysis resource file that should be used for this configuration of Filibuster.

        When supplied, analysisConfigurationFile() and analysisFile() is ignored.

        When not specified, a default analysis file will be generated to test common faults for Google's gRPC. (i.e., DEADLINE_EXCEEDED, UNAVAILABLE)

        Returns:
        absolute path of the analysis file to use.
        Default:
        ""

      • analysisConfigurationFile

        java.lang.Class<? extends FilibusterAnalysisConfigurationFile> analysisConfigurationFile
        Analysis configuration file for Filibuster.

        Will be written out to a file and supplied as a command line argument to the Filibuster server.

        When supplied, will be used to override the default analysis. When analysisFile() used, it is ignored.

        Returns:
        custom analysis configuration file.
        Default:
        cloud.filibuster.junit.configuration.examples.FilibusterDefaultAnalysisConfigurationFile.class

      • serverBackend

        java.lang.Class<? extends FilibusterServerBackend> serverBackend
        Server backend to use for Filibuster.
        Returns:
        server backend module.
        Default:
        cloud.filibuster.junit.server.backends.FilibusterLocalServerBackend.class

      • dockerImageName

        java.lang.String dockerImageName
        Docker image to use for the Docker server backend.
        Returns:
        string of image name
        Default:
        ""

      • abortOnFirstFailure

        boolean abortOnFirstFailure
        Should we abort on first failure?
        Returns:
        true
        Default:
        false

      • degradeWhenServerInitializationFails

        boolean degradeWhenServerInitializationFails
        If the Filibuster server is unavailable, should the system degrade to just running fault-free tests?
        Returns:
        boolean
        Default:
        false

      • expected

        java.lang.Class<? extends java.lang.Throwable> expected
        Whether we expect this all generated tests to throw this exception.
        Returns:
        throwable or runtime exception
        Default:
        cloud.filibuster.exceptions.filibuster.FilibusterNoopException.class

      • searchStrategy

        FilibusterSearchStrategy searchStrategy
        Which search strategy should Filibuster use?
        Returns:
        search strategy
        Default:
        cloud.filibuster.junit.FilibusterSearchStrategy.DEFAULT

      • latencyProfile

        java.lang.Class<? extends FilibusterLatencyProfile> latencyProfile
        Latency profile for the execution.
        Returns:
        latency profile implementing class.
        Default:
        cloud.filibuster.junit.server.latency.FilibusterNoLatencyProfile.class

      • serviceProfilesPath

        java.lang.String serviceProfilesPath
        Directory containing service profiles.
        Returns:
        directory
        Default:
        ""

      • serviceProfileBehavior

        ServiceProfileBehavior serviceProfileBehavior
        How service profiles should be used.
        Returns:
        service profile behavior
        Default:
        cloud.filibuster.junit.server.core.profiles.ServiceProfileBehavior.NONE

      • faultInjectionFilter

        java.lang.Class<? extends FilibusterFaultInjectionFilter> faultInjectionFilter
        Fault injection filter.
        Returns:
        filter
        Default:
        cloud.filibuster.junit.filters.NoopFilter.class

      • failIfFaultNotInjected

        boolean failIfFaultNotInjected
        Fail the test if a fault is not injected.
        Returns:
        whether test should fail.
        Default:
        false

      • failIfFaultInjectionMismatch

        boolean failIfFaultInjectionMismatch
        Fail the test if the fault at the callsite was not the fault that was injected.
        Returns:
        whether test should fail.
        Default:
        false

      • failIfFaultNotInjectedAndATrackedMethodIsInvoked

        boolean failIfFaultNotInjectedAndATrackedMethodIsInvoked
        Fail the test if a fault is not injected and a tracked method is invoked.
        Returns:
        whether service should fail.
        Default:
        false