How we test robot vacuums at ZDNET in 2024

Robot vacuums have become indispensable in the cleaning routines of many households — an estimated 14% of Americans now own one. You don’t have to live in the Jetsons to have a robot roaming around your home cleaning up after your kids and pets. 

Though iRobot was a groundbreaking robot vacuum brand in the US, the market has grown exponentially in the past five years. In an attempt to keep up with increasing consumer demand, new and existing vacuum brands have entered the robotics market, reaching the point of saturation. This has made it harder to choose a device when shopping for a robot vacuum. 

Also: The best robot vacuums for 2024: Expert tested and reviewed

Choosing a brand and a robot vacuum can be challenging, with choices of self-washing mops, strong suction power, self-emptying dustbins, and other extra accessories and features. Our mission at ZDNET is to break down which features are best suited for which people and which vacuums are the best on the market through comprehensive testing and research. 

How we test robot vacuums in 2024

ZDNET has several staff reviewers testing robot vacuums to deliver clear, unbiased reviews and information that helps readers choose the best robot vacuum for their needs. To date, we’ve tested around 30 robot vacuums, including robot vacuum mop combos. 

As a smart home enthusiast and robot vacuum fanatic, I enjoy testing robot vacuums in my home, where I can expose these devices to different environments and phenomena, mostly triggered by my dog or young kiddos. Simultaneously, ZDNET has a separate laboratory to conduct controlled testing and evaluation. This guide will cover how we test robot vacuums in each environment.

What makes a robot vacuum ZDNET recommended?

At ZDNET, we only recommend robot vacuums that we’ve tested ourselves, so our team recommends each one based on real-world experience. For a robot vacuum to be ZDNET-recommended, we expect it to meet several requirements:

• Performance: We consider features like suction power and how effectively the robot vacuums up debris on different floor types. Good performance also includes evaluating its navigation and mapping systems, including obstacle avoidance. We assess whether the battery life is sufficient to clean the desired areas on a single charge or if the robot features automatic recharge-and-resume capabilities.
• User experience: The user experience is central to enjoying a device, from unboxing to long-term use. At ZDNET, we evaluate whether a device features simple setup, use, and maintenance and the best audience for which each robot vacuum is best suited. 
• Versatility and features: Manufacturers often make promises to make their robots stand out from the competition, but delivering on these promises is not easy. We evaluate whether a device delivers on its specifications, including how well its mopping feature works, whether it truly is ‘the most powerful suction on the market,’ how it cleans tight spaces, and other unique differentiating features.
• Device longevity: Because robot vacuums are expensive, we know they’re not impulse buys—instead, they’re carefully thought-out purchases. By testing these robots for extended periods of time, we assess whether they deliver consistent performance with minimal issues in the long term, durability, and user maintenance requirements.

Unboxing process

From beginning to end, my in-home robot vacuum testing process involves constant note-taking and visual documentation with photos and videos. I begin by unboxing the robot vacuum and taking photos and notes of the process, including the setup experience, packaging quality, and accessories. 

I evaluate the robot’s build quality and design and assess whether it has extra features that make it stand out from competitors. These could include a self-emptying handheld vacuum, an extendable mop pad, or a sweeper brush. 

Setting up a testing environment

Because my dog stays on the first floor of my home, I tend to conduct most of my testing there. I do this to evaluate how the robots perform when collecting pet hair and dust and how they clean up the kitchen. 

The first floor of my home also has different floor types, including tile, hardwood, and short- and medium-pile carpet. 

I also have a 10-foot by 5-foot enclosure on my carpet and hardwood to evaluate the patterns vacuums use to clean up. Robot vacuum users prefer a linear cleaning method, where the robot navigates the area in lines, ensuring no spots are missed and leaving carpet lines that indicate the floor’s just been vacuumed.

However, some robot vacuums clean in different patterns or have different mapping procedures, so I like to see how an individual robot vacuum navigates my enclosure and how fast it does it to report any differences with competitors.

Performance testing and metrics

Once the robot is set up, I let it run through the house to create a space map. To do this, it’s best to clean up items from the floor and place furniture where it’s supposed to go to ensure the robot creates an accurate map. 

Mapping

Mapping can involve running the robot for a mapping round where it vacuums your entire floor, which can take up to about an hour, or it may just roll around creating a map with its LiDAR sensors, which should only take a few minutes. 

Testing suction power

Because I take notes and photos throughout the process, I evaluate any promises made by the manufacturer and assess whether the robot delivers on them. If a robot vacuum promises the strongest suction on the market, I see how it lives up to this claim.

I use different debris types to see how the robot suctions each up. I’ve found glow-in-the-dark sand that I can spread on different floor types to compare how much the robot picks up visually. I also use a kitchen scale to weigh how much sand or crumbs I spread and then compare that weight to what the robot picked up. 

Common debris in my testing products includes kinetic sand, coffee grounds, cracker crumbs, Cheerios, pet hair, and kibble. 

Navigating obstacles

Testing robot vacuums also involves watching how they navigate around corners, furniture, and tight spaces and how they avoid obstacles. While some robot vacuums may struggle to move around furniture or under couches, others struggle to avoid large debris on the floor and get their brush roller jammed. 

Here’s the different technology that robot vacuums use for obstacle avoidance:

• Bump sensors: Some simpler robot vacuums have bump sensors that detect physical contact with objects. When these robots bump into a chair leg, they know to change direction to navigate around it. 
• Infrared sensors: These sensors are a basic obstacle avoidance method and a step up from bump sensors. Infrared sensors emit infrared light and measure the reflection to calculate the distance from objects to avoid collisions. These sensors are also commonly used to detect stairs and keep the robot from falling.
• VSLAM sensors: Visual simultaneous localization and mapping (VSLAM) technology uses cameras to map the robot’s cleaning area. The images captured by the cameras are processed with algorithms to understand the area’s layout.
• LiDAR sensors: The Light detection and ranging (LiDAR) type of sensor creates an accurate, high-resolution map of the area, reaching high levels of detail. LiDAR sensors emit pulses of laser light and measure the time it takes for the laser to bounce back after hitting an object. The robot then calculates the distance based on the time delay and creates a precise 3D map of its surroundings.

Some robot vacuums combine artificial intelligence (AI) algorithms with their sensor array to map out obstacles on the floor, using image recognition to identify them.

During testing, I closely monitor how robot vacuums navigate furniture and tight spaces and whether they know to avoid small ones. Having to pick up every single item or cable from the floor before running a robot vacuum detracts from the convenience one would expect from such a device. I prefer to have my robot vacuum avoid my wayward charging cable rather than have it get stuck on it and cut its cleaning short while I’m out of the house.

If a robot promises to avoid obstacles strewn around the floor, I ensure it delivers. To test it, I use larger debris, like socks, wadded-up paper towel sheets, cables, small pieces of paper, small toys, leaves, and other items.

Mopping feature

A robot vacuum with a bonus mopping feature is a great way to keep your floors clean and shining. Unfortunately, some promise more than they can deliver. 

During testing, I observe how the robot mops and how effectively it removes stains on hard floors, like dry mud and coffee. After the robot is done, I ensure to check the floors to see whether it leaves streaks or residue. 

Robots tend to mop superficially. Many drag a microfiber pad across the floor that may or may not vibrate, while others have two rotating microfiber mop pads. Many may return to the base station periodically to scrub their mop pads, then return to the cleaning task. But much like if a person were dragging a wet wipe or microfiber cloth across dirty floors, the pad will inevitably get too dirty to clean effectively — the same happens with robot mops.

More recently, robot mops have evolved into mop rollers, which spin inside the robot and are scrubbed with clean water while the dirty water is kept separately. This last modality ensures the robot only mops with clean water. 

Also: The best robot vacuum mops: Expert tested and reviewed

ZDNET’s lab testing

Along with my testing in my home environment, we also test nearly every robot vacuum at the ZDNET lab in Louisville, Ky., for a more controlled experience. 

At the lab, the team unboxes the robot vacuum, measures the nozzle’s width, and weighs the vacuum’s dustbin. The lab has a testing rig with a fixed surface area to sprinkle debris on and evaluate how the robot cleans it. 

Debris includes sand, black rice, and pet hair and is spread over different floor surfaces, including low-pile, medium-pile, and hard floors. 

Lab testing requires different methodologies for various debris types. Sand and black rice are assessed with weight, while pet hair inevitably gets tangled on the brush roller, so the results tend to be sight-based.

Additional features

During performance testing, I make note of any specific features that make the robot vacuum in question different from others. Different robot vacuums advertise unique features to stand out, whether it’s a square shape, exceptional obstacle avoidance, extendable arms, special mopping features, or other bonus functionalities. 

I evaluate whether these differentiating factors perform as promised and if they are truly unique on the market.

User experience

The app experience can make or break a robot vacuum, especially for users who don’t leverage smart integrations, like Amazon Alexa or Google Home, for voice control. If a user relies on the app to control their robot vacuum, the app needs to be user-friendly and reliable.

During testing, I assess how the app performs and whether it’s reliable or not. I’ve found some robots have inconsistent connection issues, making you wait too long to connect to the robot when you open the app. User-friendliness makes an app easy to navigate, but it also gives users a variety of functions and customizations to let them feel in control of their device. 

Long-term testing

I test all robot vacuums for at least a month, but sometimes, I write about my first impressions within a week or two of testing. While I test robot vacuums, I ensure that my reviews are updated as new observations come up. 

Though many good features are strikingly obvious when testing a robot, issues are more likely to arise during repeated, long-term testing. I’ve had robots suddenly stop following the map they created after months of using it, forcing me to delete the map and recreate it. After a few months of use, I’ve also had a robot present a brush roll issue, so long-term testing is important to assess how well a robot vacuum performs.

Frequently Asked Questions