Each of the MIDAS sites report different “measurement characteristics”. This could be vehicle speed or traffic flow for a lane (or carriage way).

Each measurement characteristic has their own index attribute. i.e.

<d2lm:measurementSiteRecord version="15.0" id="ABCDEFG">
  <d2lm:measurementSpecificCharacteristics index="0"/>
  <d2lm:measurementSpecificCharacteristics index="1"/>
  <d2lm:measurementSpecificCharacteristics index="2"/>
</d2lm:measurementSiteRecord>

This is significant because the index is referenced when the MIDAS live data is reported. Every lane covered by the site will have multiple <d2lm:measurementSpecificCharacteristics/> elements, each for the type of data reported as well as any classification.

In the following example traffic speed and headway is reported for a single lane:

<d2lm:measurementSpecificCharacteristics index="0">
  <d2lm:measurementSpecificCharacteristics>
    <d2lm:specificLane>lane1</d2lm:specificLane>
    <d2lm:specificMeasurementValueType>trafficSpeed</d2lm:specificMeasurementValueType>
  </d2lm:measurementSpecificCharacteristics>
</d2lm:measurementSpecificCharacteristics>
<d2lm:measurementSpecificCharacteristics index="1">
  <d2lm:measurementSpecificCharacteristics>
    <d2lm:specificLane>lane1</d2lm:specificLane>
    <d2lm:specificMeasurementValueType>trafficHeadway</d2lm:specificMeasurementValueType>
  </d2lm:measurementSpecificCharacteristics>
</d2lm:measurementSpecificCharacteristics>

Traffic flow is categorized by <d2lm:lengthCharacteristics> which matches the length of different vehicles. For example:

<d2lm:measurementSpecificCharacteristics index="3">
  <d2lm:measurementSpecificCharacteristics>
    <d2lm:specificLane>lane1</d2lm:specificLane>
    <d2lm:specificMeasurementValueType>trafficFlow</d2lm:specificMeasurementValueType>
    <d2lm:specificVehicleCharacteristics>
      <d2lm:lengthCharacteristic>
        <d2lm:comparisonOperator>lessThanOrEqualTo</d2lm:comparisonOperator>
        <d2lm:vehicleLength>5.2</d2lm:vehicleLength>
      </d2lm:lengthCharacteristic>
    </d2lm:specificVehicleCharacteristics>
  </d2lm:measurementSpecificCharacteristics>
</d2lm:measurementSpecificCharacteristics>

Multiple <d2lm:lengthCharacteristic/> conditions can be specified for each characteristic, which we can assume matches different types of vehicles.

  • lessThanOrEqualTo 5.2
  • greaterThan 5.2 and lessThanOrEqualTo 6.6
  • greaterThan 6.6 and lessThanOrEqualTo 11.6
  • greaterThan 11.6

This would allow us to analyse the flow of traffic for different types of vehicle (cars, lorries, etc).

We want to flatten the DATEX2 locations into CSV, where there is a row for every <d2lm:measurementSpecificCharacteristics/>.

E.g.

site,site and lane,name,characteristic,lane,latitude,longitude,link,distance along link,first traffic classification condition
BAAE334664FB4C86B7DDC8A62FE05324,BAAE334664FB4C86B7DDC8A62FE05324-0,M25/4632M,trafficSpeed,lane1,51.3139308901129,-0.330020531095879,199131501,142,
BAAE334664FB4C86B7DDC8A62FE05324,BAAE334664FB4C86B7DDC8A62FE05324-1,M25/4632M,trafficHeadway,lane1,51.3139308901129,-0.330020531095879,199131501,142,
BAAE334664FB4C86B7DDC8A62FE05324,BAAE334664FB4C86B7DDC8A62FE05324-2,M25/4632M,trafficConcentration,lane1,51.3139308901129,-0.330020531095879,199131501,142,
BAAE334664FB4C86B7DDC8A62FE05324,BAAE334664FB4C86B7DDC8A62FE05324-3,M25/4632M,trafficFlow,lane1,51.3139308901129,-0.330020531095879,199131501,142,lessThanOrEqualTo 5.2

With xmlstarlet we need to match the XML element (with -m) we want a new row for. The column data is specified with -v.

Where there could be multiple elements that match, as is the case with <d2lm:lengthCharacteristic/>, the first element is used with [1].

The -i (or --if) option ensures that the <d2lm:lengthCharachteristic/>s are only output if the matching element exists.

xmlstarlet sel -T -N d2lm=http://datex2.eu/schema/2/2_0 \
  -t -m "//d2lm:measurementSiteTable[@id='NTIS_MIDAS_Measurement_Sites']/d2lm:measurementSiteRecord/d2lm:measurementSpecificCharacteristics" \
  -v '../@id' -o , \
  -v '../@id' -o "-" -v "@index" -o , \
  -v "../d2lm:measurementSiteIdentification" -o ',' \
  -v "d2lm:measurementSpecificCharacteristics/d2lm:specificMeasurementValueType" -o ',' \
  -v "d2lm:measurementSpecificCharacteristics/d2lm:specificLane" -o ',' \
  -v "../d2lm:measurementSiteLocation/d2lm:locationForDisplay/d2lm:latitude" -o ',' \
  -v "../d2lm:measurementSiteLocation/d2lm:locationForDisplay/d2lm:longitude" -o ',' \
  -v "../d2lm:measurementSiteLocation/d2lm:pointAlongLinearElement/d2lm:linearElement/d2lm:linearElementIdentifier" -o ',' \
  -v "../d2lm:measurementSiteLocation/d2lm:pointAlongLinearElement/d2lm:distanceAlongLinearElement/d2lm:distanceAlong" -o ',' \
  -i "d2lm:measurementSpecificCharacteristics/d2lm:specificVehicleCharacteristics/d2lm:lengthCharacteristic" \
  -v "(d2lm:measurementSpecificCharacteristics/d2lm:specificVehicleCharacteristics/d2lm:lengthCharacteristic)[1]/d2lm:comparisonOperator" -o ' ' \
  -v "(d2lm:measurementSpecificCharacteristics/d2lm:specificVehicleCharacteristics/d2lm:lengthCharacteristic)[1]/d2lm:vehicleLength" \
  -b \
  -n \
  NTISModel-MeasurementSites-2021-10-05-v15.0.xml > NTISModel-MeasurementSites-2021-10-05-v15.0.lanes.csv

Update 2021-10-14:

The live MIDAS data can be converted into CSV using a similar technique:

xmlstarlet sel -T \
  -N d2lm=http://datex2.eu/schema/2/2_0 \
  -N xsi=http://www.w3.org/2001/XMLSchema-instance \
  -t -m "//d2lm:siteMeasurements/d2lm:measuredValue/d2lm:measuredValue/d2lm:basicData" \
  -v '../../../d2lm:measurementSiteReference/@id' -o , \
  -v '../../../d2lm:measurementSiteReference/@id' -o '-' -v '../../@index' -o ',' \
  -v '../../../d2lm:measurementTimeDefault' -o ',' \
  -v '@xsi:type' -o ',' \
  -v 'd2lm:averageVehicleSpeed/d2lm:speed' -o ',' \
  -v 'd2lm:averageTimeHeadway/d2lm:duration' -o ',' \
  -v 'd2lm:occupancy/d2lm:percentage' -o ',' \
  -v 'd2lm:vehicleFlow/d2lm:vehicleFlowRate' -o ',' \
  -n DATEX2.xml