Samples

Basic Concepts

A quality verification is a test run based on a specific quality specification. A quality specification holds one or more quality conditions. Each quality condition represents a test algorithm configured for one or more specific datasets.

In order to execute a verification the prosuite.verification.service class is used to create the communication channel to the server and to start the verification of a specific quality specification.

service = prosuite.verification.service(host_name='localhost', port_nr=5151)
service.verify(specification=my_specification)

A quality verification can be based on an XML specification (exported from the ProSuite Data Dictionary) or on a specification created in code, containing a list of quality conditions. Quality conditions are created with the factory class prosuite.factories.quality_conditions.Conditions which contains all available test algorithms. Intellisense provides the method parameters and help/docstrings but for an overview over the available tests or a general introduction, refer to the ProSuite HTML help or the ‘Quick Reference’.

Before running the verification in Python, make sure the server is running, for example by starting prosuite-qa-microservice.exe. By default the communication channel is http://localhost:5151.

Verify a specification created in code

The Specification class holds a set of conditions that can be configured programmatically.

  1. Define the Data Model (= workspace)

  2. Create the Datasets (= feature classes, tables with optional filter) in the model

  3. Create a Service instance containing the connection properties

  4. Define the specification: create a prosuite.quality instance

  5. Create Condition instances using the static Conditions class and add them to the specification

  6. Optionally define the verification perimeter

  7. Optionally define the verification output directory

  8. Execute the verification

 1 import prosuite as ps
 2
 3 model = ps.Model("TopoModel", "D:\Test Data\ExtractStGallen.gdb") # From "ProSuite Documentation / SampleData.zip"
 4 datasets = [ps.Dataset("TLM_FLIESSGEWAESSER", model),
 5             ps.Dataset("TLM_STRASSE", model)]
 6
 7 service = ps.Service(host_name='localhost', port_nr=5151) # You might want to change this to the host and port of your ProSuite installation
 8
 9 simpleSpecification = ps.Specification(
10     name='MinimumLengthSpecification',
11     description='A very simple quality specification checking feature and segment length of roads and rivers')
12
13 for dataset in datasets:
14     simpleSpecification.add_condition(ps.Conditions.qa_min_length_0(dataset, limit=10, is3_d=False))
15     simpleSpecification.add_condition(ps.Conditions.qa_segment_length_0(dataset, 1.5, False))
16
17 envelope = ps.EnvelopePerimeter(x_min=2750673, y_min=1215551, x_max=2765845, y_max=1206640)
18
19 out_dir = 'C:/temp/verification_output' # You might want to change this to a directory that exists on your system, also make sure no Issue.gdb exists in this directory
20
21 verification_responses = service.verify(specification=simpleSpecification, output_dir=out_dir, perimeter=envelope)
22
23 for verification_response in verification_responses:
24     print(verification_response.message)
Response Messages
 1Creating external issue file geodatabase
 2Starting quality verification using quality specification MinimumLengthSpecification with verification tile size 5000Extent: 15172 x 8911
 3X-Min: 2750673
 4Y-Min: 1206640
 5X-Max: 2765845
 6Y-Max: 1215551
 7
 8Verifying quality conditions per cached tiles (container tests)
 9Processing tile 0 of 8: XMin: 2’750’673.00 YMin: 1’206’640.00 XMax: 2’755’673.00 YMax: 1’211’640.00
10Processing tile 1 of 8: XMin: 2’755’673.00 YMin: 1’206’640.00 XMax: 2’760’673.00 YMax: 1’211’640.00
11Processing tile 2 of 8: XMin: 2’760’673.00 YMin: 1’206’640.00 XMax: 2’765’673.00 YMax: 1’211’640.00
12Processing tile 3 of 8: XMin: 2’765’673.00 YMin: 1’206’640.00 XMax: 2’765’845.00 YMax: 1’211’640.00
13Processing tile 4 of 8: XMin: 2’750’673.00 YMin: 1’211’640.00 XMax: 2’755’673.00 YMax: 1’215’551.00
14Processing tile 5 of 8: XMin: 2’755’673.00 YMin: 1’211’640.00 XMax: 2’760’673.00 YMax: 1’215’551.00
15Processing tile 6 of 8: XMin: 2’760’673.00 YMin: 1’211’640.00 XMax: 2’765’673.00 YMax: 1’215’551.00
16Processing tile 7 of 8: XMin: 2’765’673.00 YMin: 1’211’640.00 XMax: 2’765’845.00 YMax: 1’215’551.00
17Quality verification finishedNumber of verified datasets: 2.
18Number of verified conditions: 4
19No category
20QaMinLength(0) TLM_FLIESSGEWAESSER - errors: 290
21QaMinLength(0) TLM_STRASSE - errors: 974
22QaSegmentLength(0) TLM_FLIESSGEWAESSER - errors: 1733
23QaSegmentLength(0) TLM_STRASSE - errors: 1939
24Warning count: 0
25Error count: 4’936
26The quality specification is not fulfilled
27
28Issues written to C:\temp\verification_output\Issues.gdb
29
30Verification report written to C:\temp\verification_output\verification.xml
31Html report:
32C:\temp\verification_output\verification.html
33Quality specification report:
34C:\temp\verification_output\qualityspecification.html

Control verification with the Issue class

The Issue class can be used to control the verification process. It can be used to stop the verification process when a certain issue condition is met. A sample Python script can be found in the ProSuite/prosuite-python-sample Repository (https://github.com/ProSuite/prosuite-python-samples/blob/main/QA%20Scripting/issue_demo.py).

 1issue_allowable = True
 2
 3 for verification_response in verification_responses:
 4     if len(verification_response.issues) > 0:
 5         for issue in verification_response.issues:
 6             # Demo Prints
 7
 8             # print(issue.description)
 9             # print(issue.involved_objects)
10             # print(issue.geometry)
11             # print(issue.issue_code)
12             # print(issue.allowable)
13             # print(issue.stop_condition)
14
15             if issue.allowable is False:
16                 print(f"Not allowed issue met: {issue.description} in {issue.involved_objects[0].table_name}")
17                 print("Stopping verification")
18                 issue_allowable = False
19                 break
20
21     if issue_allowable is False:
22         break

Verification using XML Specification

  1. Create a Service instance. In this example the service runs on a remote server machine.

  2. Define the quality specification: create a XmlSpecification instance from a specification.qa.xml file.

  3. Define the verification output directory

  4. Optionally define the verification perimeter

  5. Execute the verification

 1import prosuite
 2
 3service = prosuite.verification.service(host_name='arcgis_server', port_nr=5151)
 4
 5xml_file = "\\share\QA\specifications\road_specification.qa.xml"
 6sde_file = "\\share\connection_files\production_QA_version.sde"
 7
 8xml_spec = prosuite.XmlSpecification(specification_file=xml_file,
 9                                    specification_name="Produktionsunterstuetzung",
10                                    data_source_replacements=[["ProductionModel", sde_file]])
11
12out_dir = '\\share\QA\results\verification_output'
13
14for verification_response in service.verify(specification=xml_spec, output_dir = out_dir):
15    print(verification_response.message_level)
16    print(verification_response.service_call_status)
17    print(verification_response.message)

Notes:

  • Directories: The specified paths must be accessible by the server, hence use UNC-paths.

  • Data Source Replacements: The datasets in the XML specifications use a workspace_id as reference to the database.

For each workspace_id a valid connection is required in order to access the feature class or table. If the workspace for the workspace_id is not defined in the XML file or a different workspace (e.g. a user-specific geodatabase version) should be used, the data source replacements can be defined as part of the XML specification. For each workspace_id defined at the very end of the XML file, provide a path to a file geodatabase or an sde file.

Get specification names from XmlSpecification

1import prosuite
2xml_file = 'C:/temp/road_specification.qa.xml'
3names_list = prosuite.XmlSpecification.get_specification_names(xml_file)
4print(names_list)

Verification on Secure Channel

In this example, the grpc.ssl_channel_credentials object is created by a utility method, that gets the required root certificates automatically from the windows certificate store. For advanced scenarios or credentials on a non-windows platform, see the gRPC Python docs.

1import prosuite
2ssl_credentials = prosuite.utils.get_ssl_channel_credentials()
3
4# if channel_credentials are passed to the Verification constructor, a secure channel will be established.
5service = prosuite.verification.service(host_name='localhost', port_nr=5151, channel_credentials=ssl_credentials)

Define a WKB perimeter

1import prosuite
2poly_as_hex_string = '01ee0300000100000001eb03000001000000050000004060e5e8cfd5434100c3640aa44f32410000000000000000f8065f282dd6434100c3640aa44f32410000000000000000f8065f282dd6434170d71262d64f324100000000000000004060e5e8cfd5434170d71262d64f324100000000000000004060e5e8cfd5434100c3640aa44f32410000000000000000'
3wkb_perimeter = prosuite.WkbPerimeter(bytes.fromhex(poly_as_hex_string))
4
5# the wkb_perimeter can be assigned to the perimeter parameter in verify()

Note

The variable ‘poly_as_hex_string’ is the hex string representation of a polygon or envelope. It can be produced for example from an arcpy.Geometry. Any arcpy.Geometry can be converted to WKB and encoded as hex based string:

poly_as_hex_string = arcpy_polygon_geometry.WKB.hex()

Acessing a verification response

service.verify() returns an iterable of ResponseVerification objects. It is iterable because the verification service returns a reponse stream. Hence the progress can be printed in real-time.

1for verification_response in service.verify():
2    print(verification_response.message_level)
3    print(verification_response.service_call_status)
4    print(verification_response.message)

Advanced Parameters

Optionally, change advanced verification parameters, such as the Verification tile_size (the default is 5000m)

 1import prosuite
 2
 3xml_file = 'C:/temp/road_specification.qa.xml'
 4service = prosuite.verification.service(host_name='localhost', port_nr=5151)
 5
 6xml_spec = prosuite.XmlSpecification(
 7    specification_file=xml_file, specification_name="Produktionsunterstuetzung",
 8                                     data_source_replacements=[["ProductionModel", sde_file]])
 9
10params = prosuite.verification.VerificationParameters(tile_size=10000)
11
12out_dir = 'C:/temp/verification_output'
13
14for verification_response in service.verify(specification=spec, output_dir=out_dir, parameters=params):
15    print(verification_response)
16
17for verification_response in service.verify(specification=spec, output_dir = out_dir):
18    print(verification_response)

Start and stop the local service process

If no service is constantly running and the python script should run without interaction, e.g. as a batch job, the server process can be started directly from python on the local machine. In this example, an XML specification is used.

 1import time
 2import subprocess
 3import prosuite
 4
 5# Start the service from a local server installation with the default port.
 6# It will fail and shut down immediately if another service is already serving on the same port.
 7server_process = subprocess.Popen(r"C:\ProSuite\Server\prosuite-qa-microservice.exe")
 8
 9# Alternatively, provide a host name and custom port like this:
10# server_process = subprocess.Popen(
11#     [r"C:\ProSuite\Server\prosuite-qa-microservice.exe",
12#     "--hostname", "LOCALHOST", "--port", "12345"])
13
14# Wait for the process to start, initialize the ArcGIS license and the communication channl
15time.sleep(10)
16
17service = prosuite.verification.service(host_name='LOCALHOST', port_nr=5151)
18
19xml_file = "C:/Data/specifications/road_specification.qa.xml"
20workspace = "C:/Data/TopographicData.gdb"
21
22xml_spec = prosuite.XmlSpecification(specification_file=xml_file,
23                                    specification_name="Produktionsunterstuetzung",
24                                    data_source_replacements=[["ProductionModel", workspace]])
25
26out_dir = 'C:/Temp/verification_output'
27
28for verification_response in service.verify(specification=spec, output_dir = out_dir):
29    print(verification_response)
30
31# Stop the service
32server_process.kill()