Preamble:

Matterport virtual tours have increased in popularity over the years, in part because of their accessibility and ease of use. One of the biggest selling points, at least for me as a (former?) construction project manager, is the ability to take dimensions directly from the virtual tour.

When it comes to dimensional accuracy, the camera you use can make a big difference. Matterport advertises:

• 99% accuracy for the Matterport Pro2 (MSRP $5,500CAD + taxes for the whole kit);
• 99.9% accuracy with the Leica BLK360 1st gen (MSRP $27,000CAD + taxes for the whole kit);
• and a big dropoff in accuracy for other 360 cameras.
  

I wanted to see for myself the accuracy of these claims. If I (and Matterport Service Providers) am selling a service, I’d better be sure that what I’m selling lives up to what’s advertised.

To that effect, Architects, Engineers, and construction professionals would benefit from a deeper understanding of what are the advantages and limitations of the virtual tour’s measuring capabilities.

So, how reliable are Matterport’s claims for accuracy?

I ran a few experiments on a very simple space (a square-shaped room in an open apartment) and here’s what I found.

This article covers:

• Dimensional accuracy between 3 different devices;
• Comparison of measurement values between devices;
• Comparison of measurement values within devices, based on good scanning practices;
• Making sense of your measurement value.

This article does not cover:

• Anything to do with the Matterport Pro3;
  • Yes I get it, the Pro3 is Matterport’s newest camera with Lidar, with “99.9% accuracy.” Stay tuned for future articles.
• Photo quality between devices;
• Ease / rapidity of scanning from one device to the next, or alignment errors ;
• Dimensional accuracy on Matterpak point clouds or e57 files;

Methods for 1st Experiment:

Experiment #1: Scan Positions for Each Test

Test 1
Ricoh Theta Z1: Scanned casually. 17 scan positions.

Test #2
Matterport Pro2: Done casually. 8 scan positions.

Test #3
Matterport Pro2: Sone intentionally to yield high accuracy measurements. 35 scan positions.

Test #4
Leica BLK360: Done casually. 5 scan positions.

Test #5
Leica BLK360: Done intentionally to yield high accuracy measurements. 9 scan positions.

Measurement results for each test:

Test 1
Ricoh Theta Z1: Within 2%

Test 2
Pro2: Within 1.5%

Test 3
Pro2: Within 1%

Test 4
BLK360: Within 0.2%

Test 5
BLK360: Within 0.2%

Experiment #1: Comparison of measurements (3D space vs Bosch Laser Measure)

Comparing the measurement accuracy of the Ricoh Theta Z1, the Matterport Pro2, and the Leica BLK360G1

Observations from Experiment #1 data set:

Ricoh Theta Z1

Dimensions were actually pretty good, for this small space. Within 2%.

The Ricoh Theta Z1 (and maybe other less expensive 360 cameras) did surprisingly well in terms of measurement accuracy, but only when I was using the measurement tool to manually measure start and finish points on the 3D space.

It had the most issues when trying to create measurements by using the double click method (explained earlier). An end user will have to spend a lot more time getting room measurements, unless they measure from plan view. But measuring from plan view is not as accurate as measuring from the 3D space, and measuring heights is not an option in plan view.

Ricoh Theta Z1 – An example of the Z1’s poor geometry in 3D space, making it difficult to get a measurement to the wall across the room by double clicking.

Even though I was able to get results within 2% with the Z1, it took a bit of time to get the correct start and finish points using the measuring tool, as well as an understanding of the limitations of Matterport’s platform. Most users don’t have this technical knowledge, and can lead to significant measurement errors.

Matterport Pro 2

In this 1st experiment, the Pro2 delivered 1% accuracy when enough scan positions were taken as it did in Test #3.

While it is possible to fall short of the Pro2’s 1% accuracy claim if there are too few scan positions, I was surprised how I got the same results for the Pro2’s length and width in both test cases.

For the height, however, Test #2 fell just outside of the 1% accuracy, while Test #3 just barely came within 1%. Given that the Pro2 has a distance limitation of 5m, one might predict measurement issues at higher ceiling heights.

It’s important to keep in mind the implication of what it means to be off by 1%. It may mean being off by up to 10cm (in either direction) for a distance of 10m. While this was not the case on this test (due to the simple geometry of the apartment i.e. one big square), issues may arise when there are multiple rooms and larger areas.

Leica BLK360 (and The Number Rounding Problem)

If I’m going to be perfectly honest with myself, I would score this at 0.2% accuracy.

The thing that bothered me the most about this experiment was that the 3D space measurements were only accurate to 1cm or 1″. I’m assuming any dimension taken from the 3D space has to be rounded to the nearest centimeter (in metric), and doing so affects the accuracy results, specifically for the Leica BLK360.

Why is this a problem? Here’s an example:

• The width of the apartment was measured with a laser to be 9.323m;
• Test #5 had a measured width of 9.31m. If the measured BLK360 result was rounded to 2 decimal places, the measurement before rounding could have been anywhere between 9.305m and 9.314m.
  • If it were rounded down from 9.314mm, then the actual % difference in measurement results would be -0.099% (or 9mm);
  • If it were rounded up from 9.305mm, then the actual % difference in measurement results would be -0.195% (or 18mm);

This leaves the possibility that Leica BLK360 may be, in some instances, only accurate to about 0.2% (or worse, as we’ll see later).

Also, as the actual measured dimensions get smaller (smaller rooms, width of corridors, etc.), rounding to the nearest centimeter creates an even greater % difference between the measured result and the actual result.

Experiment #2: Performance at heights / Beam to Beam

Both the Ricoh Theta Z1 and the Matterport Pro2 performed well enough when measuring the length of the apartment, where the wall-to-wall measurements were at about 1.25m from the ground.

The measurement results started to suffer when I had them do a wall-to wall measurement at a height of 2.9m from the ground, where the closest scan positions to the wall might have been further from the cameras. The BLK360, however, did not have these height issues.

Seeing as the measurement data only gets worse for cameras when measuring at a distance, I need to know just how bad it can get.

This next experiment was for measuring the distance between the bottom part of the beams in the ceiling of the apartment. The vertical distance between the top of the Z1, Pro2, and BLK360 and the beam were about 2.2m. From their respective 3D spaces, the measured distances were compared to the results from the laser scanner.

* Remember, the Matterport Virtual Tour rounds measurements to 2 decimal places. There is a possibility of being off by an additional 5mm. If this is the case, then these additional 5mm would account for a greater significance in measurement (in)accuracy for the BLK360 when the claim is “0.1% accuracy” compared to the Pro2’s “1%”.

Observations from Experiment #2 data set:

All capture devices had diminished accuracy when measuring elements in the ceiling, at a distance of 2.2m away.

Ricoh Theta Z1

Not a big deal for the Z1 (or other 360 cameras), since the expectations for accuracy were within 4% – 8%;

Matterport Pro2

This is a bigger deal for the Pro2, leaving the “within 1%” accuracy range. It would be safe to estimate the Pro2’s accuracy to within 2% for ceilings that are 2m away from the camera. The range of the Pro2 is reported to be 5m, so I’m not sure how quickly the results will deteriorate at distances > 2m.

Leica BLK360

Ooof. These results suffer the most, with a potential of having measurement results being down to within 0.5% accuracy due to number rounding… a far cry from 0.1%.

Experiment #3: Measuring the pipes in the ceiling

Experiment #3 was for measuring the distance between a set of pipes that are visible in the ceiling. This is helpful for Mechanical Engineers, who are interested in using these measurements for planning their work.

For this experiment I used a tape measure (more appropriate + more commonly used for this use case) and compared to the results from the Z1, Pro2, and BLK360.

Ricoh Theta Z1 – the measuring tool doesn’t really distinguish the difference between the pipe and the beam just above it. It’s as if it’s looking at a flat surface. No good.

Matterport Pro2 – the measuring tool gets a sense that there is something different between the pipe and the beam above it, but does a poor job of snapping to the desired point. The comparison of the measurement start and finish points tells the story. This is pretty bad, especially if Pro2 users / clients are relying on this tool for measurement data.

Leica BLK360 – Here we can see how the measurement tool (the curser) follows along the contour of the pipe, clearly distinguishing the limits between the pipe and the beam above it.

The BLK360 reigns supreme for measuring piping in ceilings.

Observations from Experiment #3 data set:

It’s pretty clear – the Leica BLK360 is the only scanning device that can accurately detect and measure piping. No other devices in this experiment have any business attempting that in a professional setting.

Considering the lackluster performance of the Pro2 for measuring the pipes in the ceiling, it raises a concern about the quality of the data from the Matterpaks, e57s, and other digital assets that are being sold for the purpose for AEC (Architecture, Engineering, Construction),

Final Thoughts

However, should AEC professional require 0.1% and greater measurement accuracies in the planning phases of a project, one can always consider registering the Leica BLK360 point cloud in Leica’s Cyclone app. Compared to the Matterport’s 3D space (and e57 files / matterpak files), the same BLK360 data, when registered in Cyclone and viewed in AutoDesk Recap, will attain the 0.1% measurement tolerance… but that’s outside the scope of this article. Stay tuned.

Properly registered Cyclone point cloud, using scan data from BLK360, as seen in Autodesk Recap, with a comparison of measurement data. Now we’re within 0.1% accuracy.