1 Creating entity
- 1.1 Flaky test example
- 1.2 Waiting for the READ database to be updated
2 Updating entity
3 Deleting entity
4 Sign off

Based on daily test run statistics, we’ve found that most of failures happen because of READ and WRITE databases synchronization,

when test tries to interact (read, update, delete) with newly created (updated, deleted) entity before READ database updated and as a result test fails with NotFound status code from the API.

To avoid writing flaky API tests there’re are some recommendations below:

Creating entity

create request model
send create request
assert response
assert entity is accessible via read API

Flaky test example

Firstly, let’s have a look at the flaky test checking creation of the new user in the application:

Feature class	Context class

Feature class

Context class

 [Scenario(DisplayName = "Create new user")]
        public void Create_new_user()
        {
            Runner.WithContext<UserContext>().RunScenario(
                _ => _.Given_user_model(),
                _ => _.When_send_create_user_request(),
                _ => _.Then_response_code_is(HttpStatusCode.Created),
                _ => _.When_send_get_user_request(),
                _ => _.Then_response_code_is(HttpStatusCode.OK)
            );
        }

        public void Given_user_model()
        {
            _userModel = new UserModel
            {
                Email = $"{Fixture.Create<string>()}@gmail.com",
                FirstName = Fixture.Create<string>(),
                LastName = Fixture.Create<string>(),
                ProviderUserId = Fixture.Create<string>(),
                UserName = Fixture.Create<string>()
            };
        }
        
         public void When_send_create_user_request()
        {
            _apiResponse = _apiClient.CreateUser(_userModel.AsJson());
            _userId = _apiResponse.GetGlobalId();
        }
        
         public void Then_response_code_is(HttpStatusCode expectedCode)
        {
            AssertResponseCodeIs(_apiResponse, expectedCode);
        }
        
         public void When_send_get_user_request()
        {
            _apiResponse = _apiClient.GetAUser(_userId);
        }
        
        public void Then_response_code_is(HttpStatusCode expectedCode)
        {
            AssertResponseCodeIs(_apiResponse, expectedCode);
        }

The issue is in the lines 8 and 9 in the test scenario

 _ => _.When_send_get_user_request(),
 _ => _.Then_response_code_is(HttpStatusCode.OK)

Sooner or later this test will fail with API response status code: 404 Not Found

This happens because by the moment we send GET request newly created user is not in the READ database yet.

Waiting for the READ database to be updated

The solution is to wait util READ database updated with new user before sending the GET request and checking the status code

We have at least two implementation of “waiting“ mechanism:

Wait of some fixed amount of time:

Task.Delay(timeSpan).Wait();
Thread.Sleep(timeSpan);

…

The issue with this approach is that we don’t know exactly how much time needed to wait in every particular test, considering waiting time depends on the environment, network loading etc.

Also we can’t set, for example,10 seconds timeout in one place, as a global variable for every test, because it will significantly slow down the whole test suite execution time,

that’s why we will use another way of waiting mechanism implementation - polling mechanism

2. Waiting with polling interval

In this case we’ll send GET request in cycle with a response checking after each iteration

            for (var i = 0; i < iterationsCount; i++)
            {
                actualResponse = request.Invoke();
                if (condition.Invoke(actualResponse))
                {
                    break;
                }
               Task.Delay(frequency);
            }

Here we have waiting with a fixed amount of time: Task.Delay(frequency); but in this case waiting time is configurable, much more smaller and give tests more flexibility

The logic in the code snippet above was wrapped into ExecuteUntilResponseIs method in the ApiClientUtils class

Now let’s have a look how test for user creation should be implemented to make it stable and independent from database synchronization

We need to replace two lines:

 _ => _.When_send_get_user_request(),
 _ => _.Then_response_code_is(HttpStatusCode.OK)

with:

 _ => _.And_get_request_returns(HttpStatusCode.OK)

Feature class	Context class

Feature class	Context class
`[Scenario(DisplayName = "Create new user")] public void Create_new_user() { Runner.WithContext<UserContext>().RunScenario( _ => _.Given_user_model(), _ => _.When_send_create_user_request(), _ => _.Then_response_code_is(HttpStatusCode.Created), _ => _.And_get_a_user_request_returns(HttpStatusCode.OK) ); }`	`public void And_get_a_user_request_returns(HttpStatusCode expectedCode) { var apiResponse = ApiClientUtils.ExecuteUntilResponseIs( () => _apiClient.GetAUser(_userId), r => r.StatusCode == expectedCode, frequency: 2, timeoutSec: 10); AssertResponseCodeIs(apiResponse, expectedCode, $"User with globalId {_userId} wasn't created!"); }`

In this case we’ll send request to find user by global Id every 2 seconds , with a 10 seconds timeout

and after every request execution we check the response status code, if matches - exit the cycle and return API response

And finally after small refactoring moving lines 3-5 into separate method we get:

Feature class	Context class

Feature class	Context class
`[Scenario(DisplayName="Createnewuser")] publicvoidCreate_new_user() { Runner.WithContext<UserContext>().RunScenario( _=>_.Given_user_model(), _=>_.When_send_create_user_request(), _=>_.Then_response_code_is(HttpStatusCode.Created), _=>_.And_get_a_user_request_returns(HttpStatusCode.OK) ); }`	`public void And_get_a_user_request_returns(HttpStatusCode expectedCode) { AssertRequestReturnsStatusCode( () => _apiClient.GetAUser(_userId), expectedCode, $"User with globalId {_userId} wasn't created!"); }`

AssertRequestReturnsStatusCode method comes from BaseContext class.

Updating entity

create request model
send create request
assert entity is accessible via read API
send update request
assert response status code
send get request
assert entity was updated

Flaky test example

Let’s have a look at flaky test:

Feature class	Context class

Feature class

Context class

 [Scenario(DisplayName = "Update user")]
        public void Update_user()
        {
            const string name = "new name";

            Runner.WithContext<UserContext>().RunScenario(
                _ => _.Given_user_exists(),
                _ => _.When_user_name_updated(name),
                _ => _.Then_response_code_is(HttpStatusCode.NoContent),
                _ => _.When_send_get_user_request(),
                _ => _.Then_user_name_updated(name)
            );
        }

        public void Given_user_exists()
        {
            Given_user_model();
            When_send_create_user_request();
            Then_response_code_is(HttpStatusCode.Created);
            And_get_a_user_request_returns(HttpStatusCode.OK);
        }
      
        public void When_user_name_updated(string name)
        {
            _userModel.UserName = name;

            _apiResponse =  _apiClient.UpdateUser(_userId, _userModel.AsJson());
        }
        
        public void Then_user_name_updated(string expectedName)
        {
            var actualName = _apiResponse.AsJson<UserGetModel>().UserName;

            AssertionUtils.AssertEqual(expectedName, actualName, 
                $"User name wasn't updated!\n" +
                $"Api response: {_apiResponse.Content}");
        }

Here we have issue in these two scenario steps:

 _ => _.When_send_get_user_request(),
 _ => _.Then_user_name_updated(name)

Considering GET request may be executed before READ database updated, variable “actualName” may have the old value and test will fail.

We need to modify Then_user_name_updated step and first and straightforward solution looks like this:

Code	Comments

Code

Comments

public void Then_user_name_updated(string expectedName)
        {
            var apiResponse = ApiClientUtils.ExecuteUntilResponseIs(
                () => _apiClient.GetAUser(_userId),
                r => r.AsJson<UserGetModel>().UserName.Equals(expectedName));

            var actualName = apiResponse.AsJson<UserGetModel>().UserName;

            AssertionUtils.AssertEqual(expectedName, actualName, 
                $"User name wasn't updated!\n" +
                $"Api response: {_apiResponse.Content}");
        }

In this case we send request in cycle and wait until UserName in the response equals to the expectedName

After small refactoring we get:

Code	Comments

Code

Comments

public void Then_user_name_updated(string expectedName)
        {
            bool expectedCondition(IRestResponse r) => r.AsJson<UserGetModel>().UserName.Equals(expectedName);

            var apiResponse = ApiClientUtils.ExecuteUntilResponseIs(
            () => _apiClient.GetAUser(_userId), expectedCondition, frequency: 2, timeoutSec: 10);

           Assert.True(expectedCondition(apiResponse), $"User name wasn't updated!\n" +
                                                       $"Api response: {apiResponse.Content}");
        }

This is satisfactory solution for checking single field was updated in the model

Working with multiple fields

Usually we need update multiple fields, let’s have a look how to deal with that

Feature class	Context class

Feature class

Context class

[Scenario(DisplayName = "Update user, multiple fields")]
        public void Update_user_multiple_fields()
        {
            Runner.WithContext<UserContext>().RunScenario(
                _ => _.Given_user_exists(),
                _ => _.When_user_model_updated_with_user_name("new user name"),
                _ => _.When_user_model_updated_with_first_name("new first name"),
                _ => _.When_user_model_updated_with_last_name("new last name"),
                _ => _.When_send_update_user_request(),
                _ => _.Then_response_code_is(HttpStatusCode.NoContent),
                _ => _.Then_user_updated()
            );
        }

        public void When_user_model_updated_with_user_name(string name)
        {
            _userModel.UserName = name;
        }

        public void When_user_model_updated_with_first_name(string firstName)
        {
            _userModel.FirstName = firstName;
        }

        public void When_user_model_updated_with_last_name(string lastName)
        {
            _userModel.LastName = lastName;
        }

        public void When_send_update_user_request()
        {
            _apiResponse = _apiClient.UpdateUser(_userId, _userModel.AsJson());
        }

In the step Then_user_updated we need to check that API response contains the User entity with username, first and last names having new values, but there’re some challenges:

Firstly, we may have more then 3 values to compare (entity may consist of 10 and more fields)

Secondly, we will need to check as many fields in the API response as we updated in the test scenario

 bool expectedCondition(IRestResponse r) =>
                r.AsJson<UserGetModel>().UserName.Equals(expectedValue)
                && r.AsJson<UserGetModel>().FirstName.Equals(expectedValue)
                && r.AsJson<UserGetModel>().UserName.Equals(expectedValue);

This may be OK for a couple of fields but not for 10 and above, the code becomes hard for reading and maintaining (not mentioning the code writing rules)

Using “Equals“ method for objects comparison

The better way to compare two models instead of comparing by particular fields is using “Equals” method.

But before test updating, let’s have a small example of how “Equals“ method works

private class Rectangle
        {
            public int A { get; set; }
            public int B { get; set; }
        }

        [Fact]
        public void Test()
        {
            var rect1 = new Rectangle {A = 1, B = 2};
            var rect2 = new Rectangle { A = 1, B = 2 };

            var isEqual = rect1.Equals(rect2); // false

            Assert.Equal(rect1, rect2); // fails with en error: Message: Assert.Equal() Failure
                                        //                      Expected: Rectangle { A = 1, B = 2 }
                                        //                      Actual:   Rectangle { A = 1, B = 2 }
        }
        
        [Fact]
        public void Test2()
        {
            var rect1 = new Rectangle { A = 1, B = 2 };
            var rect2 =rect1;

            var isEqual = rect1.Equals(rect2); // true
        }

As you can see, by default, “Equals” method just checks weather two variables refer to the same instance of Rectangle class

To “say“ to “Equals“ method that you want to check not references but values you need to override “Equals” and “GetHashCode“ methods for the target class.

private class Rectangle
        {
            public int A { get; set; }
            public int B { get; set; }

            private bool Equals(Rectangle other)
            {
                return A == other.A && B == other.B;
            }

            public override bool Equals(object obj)
            {
                if (ReferenceEquals(null, obj)) return false;
                if (ReferenceEquals(this, obj)) return true;
                if (obj.GetType() != this.GetType()) return false;
                return Equals((Rectangle) obj);
            }

            public override int GetHashCode()
            {
                unchecked
                {
                    return (A * 397) ^ B;
                }
            }
        }

Visual Studio has inbuilt tools to help in overriding “Equals” and “GetHashCode“ methods.

Open “Generate code“ window (usually it’s buttons: Alt+Ins)

2. Select members of the class we will use for comparison of two objects and click “Finish“ button

“Equals” and “GetHashCode“ methods will be generated automatically and now we’re ready can compare two objects:

private class Rectangle
        {
            public int A { get; set; }
            public int B { get; set; }
        }

        [Fact]
        public void Test()
        {
            var rect1 = new Rectangle {A = 1, B = 2};
            var rect2 = new Rectangle { A = 1, B = 2 };

            var isEqual = rect1.Equals(rect2); // true
        }

Now let’s do the same for the UserModel class:

Feature class	Context class

Feature class

Context class

public class UserModel
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string UserName { get; set; }
        public string Email { get; set; }
        public string ProviderUserId { get; set; }
        public int Status { get; set; }
    }

public class UserModel
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string UserName { get; set; }
        public string Email { get; set; }
        public string ProviderUserId { get; set; }
        public int Status { get; set; }

        protected bool Equals(UserModel other)
        {
            return string.Equals(FirstName, other.FirstName) && string.Equals(LastName, other.LastName) && string.Equals(UserName, other.UserName) && string.Equals(Email, other.Email) && Status == other.Status;
        }

        public override bool Equals(object obj)
        {
            if (ReferenceEquals(null, obj)) return false;
            if (ReferenceEquals(this, obj)) return true;
            if (obj.GetType() != this.GetType()) return false;
            return Equals((UserModel) obj);
        }

        public override int GetHashCode()
        {
            unchecked
            {
                var hashCode = (FirstName != null ? FirstName.GetHashCode() : 0);
                hashCode = (hashCode * 397) ^ (LastName != null ? LastName.GetHashCode() : 0);
                hashCode = (hashCode * 397) ^ (UserName != null ? UserName.GetHashCode() : 0);
                hashCode = (hashCode * 397) ^ (Email != null ? Email.GetHashCode() : 0);
                hashCode = (hashCode * 397) ^ Status;
                return hashCode;
            }
        }
    }

Test with Then_user_updated implementation will look like this:

Feature class	Context class

Feature class

Context class

 [Scenario(DisplayName = "Update user, multiple fields")]
        public void Update_user_multiple_fields()
        {
            Runner.WithContext<UserContext>().RunScenario(
                _ => _.Given_user_exists(),
                _ => _.When_user_model_updated_with_user_name("new user name"),
                _ => _.When_user_model_updated_with_first_name("new first name"),
                _ => _.When_user_model_updated_with_last_name("new last name"),
                _ => _.When_send_update_user_request(),
                _ => _.Then_response_code_is(HttpStatusCode.NoContent),
                _ => _.Then_user_updated()
            );
        }

        public void Then_user_updated()
        {
            var apiResponse = ApiClientUtils.ExecuteUntilResponseIs(
                () => _apiClient.GetAUser(_userId), 
                r=>r.AsJson<UserModel>().Equals(_userModel), frequency: 2, timeoutSec: 10);

            var actualUser = apiResponse.AsJson<UserModel>();

            AssertionUtils.AssertEqual(_userModel, actualUser, "User wasn't updated!");
        }

In the method Then_user_updated we compare two instances of the UserModel class, first instance is _userModel (expected value), second, comes from API response of _apiClient.GetAUser(_userId) (actual value).

Implementation of Then_user_updated step may be simplified using AssertResponseContainsItem method, which comes from BaseContext class

Feature class	Context class

Feature class

Context class

 [Scenario(DisplayName = "Update user, multiple fields")]
        public void Update_user_multiple_fields()
        {
            Runner.WithContext<UserContext>().RunScenario(
                _ => _.Given_user_exists(),
                _ => _.When_user_model_updated_with_user_name("new user name"),
                _ => _.When_user_model_updated_with_first_name("new first name"),
                _ => _.When_user_model_updated_with_last_name("new last name"),
                _ => _.When_send_update_user_request(),
                _ => _.Then_response_code_is(HttpStatusCode.NoContent),
                _ => _.Then_user_updated()
            );
        }

        public void Then_user_updated()
        {
            AssertResponseContainsItem(() => _apiClient.GetAUser(_userId), _userModel);
        }

Deleting entity

create request model
send create request
assert entity is accessible via read API
send delete request
send get request
assert entity was deleted

Test checking the deleting entity looks very similar to reading one.

The main point - after sending DELETE request we need to send GET request until it returns Not_Found status code:

Feature class	Context class

Feature class

Context class

[Scenario(DisplayName = "Delete a user")]
        public void Delete_a_user()
        {
            Runner.WithContext<UserContext>().RunScenario(
                _ => _.Given_user_exists(),
                _ => _.When_send_delete_user_request(),
                _ => _.Then_response_code_is(HttpStatusCode.NoContent),
                _ => _.And_get_a_user_request_returns(HttpStatusCode.NotFound)
            );
        }

        public void Given_user_exists()
        {
            Given_user_model();
            When_send_create_user_request();
            Then_response_code_is(HttpStatusCode.Created);
            And_get_a_user_request_returns(HttpStatusCode.OK);
        }
        
        public void When_send_delete_user_request()
        {
            _apiResponse = _apiClient.DeleteUser(_userId);
        }
        
        public void Then_response_code_is(HttpStatusCode expectedCode)
        {
            AssertResponseCodeIs(_apiResponse, expectedCode);
        }
        
        public void And_get_a_user_request_returns(HttpStatusCode expectedCode)
        {
            AssertRequestReturnsStatusCode(
                () => _apiClient.GetAUser(_userId), expectedCode, $"User with globalId {_userId} wasn't created!");
        }

Future Platform Development

How to write stable API tests

Creating entity

Flaky test example

Waiting for the READ database to be updated

Updating entity

Flaky test example

Working with multiple fields

Using “Equals“ method for objects comparison

Deleting entity

Sign off