VSTS Error : No test assemblies found matching the pattern

Question

I am planning to run end to end tests overnight using VSTS. Is there any guiding document or sample plans to show how to implement this.

Basically we have API projects in VSTS where we have End to End test projects within the main solution. So we wanted to run those tests overnight. All the other tests run as part of the build but NOT E2E as it should run after the deployment.

This is the build.yaml

resources:
- repo: self
queue:
  name: Hosted VS2017
  demands: 
  - vstest
  - msbuild
  - visualstudio

variables:
  BuildPlatform: 'any cpu'
  BuildConfiguration: 'release'
steps:
- task: NuGetToolInstaller@0
  displayName: 'Use NuGet 4.4.1'
  inputs: 
    versionSpec: 4.4.1


- task: NuGetCommand@2
  displayName: 'NuGet restore'

- task: VSBuild@1
  displayName: 'Build solution'
  inputs:
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(Build.ArtifactStagingDirectory)\app.zip" /p:DeployIisAppPath="Default Web Site"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

- task: powershell@2
  inputs:
    targetType: filePath
    filePath: deploy/lib/GetSwaggerByExe.ps1
    arguments: '-RootDirectory "src/SwaggerGenerator"  -OutputFile "deploy/swagger/swagger.json" '
  displayName: 'Create Swagger file'

- task: VSTest@2
  displayName: 'Test Assemblies'
  inputs:
    testAssemblyVer2: |
     **\$(BuildConfiguration)\*test*.dll
     !**\obj\**
     !**\*\*.EndToEnd.Integration.Tests*.dll
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

- task: PublishTestResults@2
  displayName: 'Publish API Test Results'
  inputs:
    testResultsFormat: VSTest
    testResultsFiles: '**/*.trx'

- task: PublishSymbols@2
  displayName: 'Publish symbols path'
  inputs:
    PublishSymbols: false
  continueOnError: true

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact'
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)/app.zip'
    ArtifactName: app

- task: PublishBuildArtifacts@1
  displayName: 'Publish deploy scripts'
  inputs:
    PathtoPublish: deploy

    ArtifactName: deploy

I have below task in CD

Is this because of the incorrect path in test files parameter in the above task step. If so how to find out the correct path? I have referred below links as well, no luck though.

Build logs

Project "D:\a\1\s\src\XXXXSearch.Api.sln" (1) is building "D:\a\1\s\src\XXXSearch.EndToEnd.Integration.Tests\XXXSearch.EndToEnd.Integration.Tests.csproj" (6) on node 1 (default targets).
2018-12-23T09:48:50.2267347Z PrepareForBuild:
2018-12-23T09:48:50.2267390Z   Creating directory "bin\Release\".
2018-12-23T09:48:50.2269510Z   Creating directory "obj\Release\".

 Creating "D:\a\1\s\src\XXXXSearch.EndToEnd.Integration.Tests\obj\Release\XXXSearch.EndToEnd.Integration.Tests.csproj.CopyComplete" because "AlwaysCreate" was specified.
2018-12-23T09:48:50.4678280Z _CopyAppConfigFile:
2018-12-23T09:48:50.4678368Z   Copying file from "app.config" to "bin\Release\XXXSearch.EndToEnd.Integration.Tests.dll.config".
2018-12-23T09:48:50.4686244Z CopyFilesToOutputDirectory:
2018-12-23T09:48:50.4686316Z   Copying file from "obj\Release\XXXSearch.EndToEnd.Integration.Tests.dll" to "bin\Release\XXXSearch.EndToEnd.Integration.Tests.dll".
2018-12-23T09:48:50.4691372Z   XXXSearch.EndToEnd.Integration.Tests -> D:\a\1\s\src\XXXSearch.EndToEnd.Integration.Tests\bin\Release\XXXSearch.EndToEnd.Integration.Tests.dll
2018-12-23T09:48:50.4693481Z   Copying file from "obj\Release\XXXSearch.EndToEnd.Integration.Tests.pdb" to "bin\Release\XXXSearch.EndToEnd.Integration.Tests.pdb".
2018-12-23T09:48:50.4755662Z Done Building Project "D:\a\1\s\src\XXXSearch.EndToEnd.Integration.Tests\XXXSearch.EndToEnd.Integration.Tests.csproj" (default targets)

Added below tasks in build.yaml file

- task: CopyFiles@2
  inputs:
    contents: D:/a/1/s/src/XXX.SiteSearch.EndToEnd.Integration.Tests/bin/Release/XXX.SiteSearch.EndToEnd.Integration.Tests.dll
    targetFolder: $(Build.ArtifactStagingDirectory)

- task: PublishBuildArtifacts@1 
  displayName: 'Publish E2E Artifact' 
  inputs: 
    PathtoPublish: 'D:/a/1/a/src/XXX.SiteSearch.EndToEnd.Integration.Tests/bin/Release/XXX.SiteSearch.EndToEnd.Integration.Tests.dll' 
    ArtifactName: e2e

Solution-1

Solution-2

Solution-3

Daniel Mann · Accepted Answer · 2018-12-26T02:08:42.497

2

Look at your task definition for the artifacts you're publishing:

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact'
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)/app.zip'
    ArtifactName: app

- task: PublishBuildArtifacts@1
  displayName: 'Publish deploy scripts'
  inputs:
    PathtoPublish: deploy

Where are you publishing your test assemblies? Answer: You're not.

If you don't publish something as an artifact, it's not going to be available down-stream in your release.

The solution is to publish your test assemblies (and other any relevant files) as a separate artifact.

edited Dec 26 '18 at 02:08

answered Dec 25 '18 at 16:26

Daniel Mann

57,011
13
100
120

As you correctly mentioned haven't put `test assemblies` into `$(BuildConfiguration)` since don't won't to run `E2E` as part of the build as mentioned in the question. Could you please elaborate on your comment i.e. "put a step in your build to run a PowerShell script of `get-childitem $env:SYSTEM_DEFAULTWORKINGDIRECTORY -rec` " – Shabar Dec 25 '18 at 23:07
Added build logs section related to `EndToEnd` tests which shows that EndtoEnd tests `dll` copied across. So then am I getting the path incorrect on the CD task (VsTest - End2End Test)? – Shabar Dec 25 '18 at 23:47
Is there a way to add those test assemblies to existing `app.zip` mentioned in the above `build.yaml`? Other confusion I have is, in the build logs what does this mean `2018-12-23T09:48:50.4691372Z XXXSearch.EndToEnd.Integration.Tests -> D:\a\1\s\src\XXXSearch.EndToEnd.Integration.Tests\bin\Release\XXXSearch.EndToEnd.Integration.Tests.dll` Becuz still I cannot see those dll's in the app.zip file. Sorry I am pretty new to this area. – Shabar Dec 28 '18 at 05:22
1

@SMPH You don't want to add them to `app.zip`. That's for deployment; you don't deploy tests. Add another publish artifact step and publish the tests as a separate artifact. – Daniel Mann Dec 28 '18 at 06:12
added this `- task: PublishBuildArtifacts@1 displayName: 'Publish E2E Artifact' inputs: PathtoPublish: '$(Build.ArtifactStagingDirectory)/*.EndToEnd.Integration.Tests.dll' ArtifactName: e2e` but it gives this error `##[error]Publishing build artifacts failed with an error: Not found PathtoPublish: D:\a\1\a\*.EndToEnd.Integration.Tests.dll` – Shabar Dec 28 '18 at 07:51
And the other point is since `e2e-tests` has to be run after the deployment so I believe it has to be deployed first in the environment and then execute. – Shabar Dec 28 '18 at 09:34
1

@SMPH That assembly wouldn't be in `$(Build.ArtifactStagingDirectory)`, because you're not telling it to put all build outputs there, just the zip file. Look at your build outputs: `D:\a\1\s\src\XXXSearch.EndToEnd.Integration.Tests\bin\Release\XXXSearch.EndToEnd.Integration.Tests.dll`. You only need to deploy the tests if they depend on something specific to an individual server (such as a message queue). If they're just plain old Selenium/Coded UI tests or integration tests, you can run them from any private agent that can talk to the appropriate server(s). – Daniel Mann Dec 28 '18 at 14:24
@ Daniel Mann, I added required steps in `build.yaml` (added to the question) Now build is running successfully, But in `VsTest- End2End Tests` task shows this logs. `No test is available in D:\a\r1\a\SiteSearchFramework\e2e\XXX.SiteSearch.EndToEnd.Integration.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.` The issue is it's looking for `D:\a\r1\a...` path but files there in `D:\a\1\a....` Is there a way to have this test `dll` in `D:\a\r1\a` ? – Shabar Dec 29 '18 at 10:32
I have opened separate question for the above said issue as it is not directly related to this question. https://stackoverflow.com/questions/53974646/vsts-waring-make-sure-that-test-discoverer-executors-are-registered-and-plat – Shabar Dec 30 '18 at 01:27

VSTS Error : No test assemblies found matching the pattern

1 Answers1