Introduction
In 1995, when English Premier League top team Arsenal decided to buy mercurial Dutch forward Denis Bergkamp, they knew they were taking on a person with proven aviophobia. A year earlier the player had developed the fear of flying. The decision to invest meant that Bergkamp would not be able to play in overseas matches, due to his reluctance to travel by air. Still, the record-breaking club didn’t hesitate. It was a decision that would pay off immensely as Bergkamp became almost indispensable, winning three major titles for the club, hardly ever traveling overseas for matches.
Bots are no different. They are digital must-haves – too important for organizations to do without, yet too portentous to take lightly.
High on indispensability
Introduced initially to make repetitive tasks easier, without the need for human intervention, a bot – which is short for robot – is an automated software application operating over a network. The sheer benefits they confer in terms of speed, accuracy, and ability to simulate human behavior, make them almost indispensable to organizations. Searchenginejournal (3) puts it succinctly: ‘You don’t want to block Google (a bot). You very much want it to crawl and index your website. Can you imagine what would happen to your search engine rankings?’
Some of the more common ones (1) that are productively and widely used are:
- Chatbots that are used by customer support teams to answer basic queries from visitors to a website. Often given a ‘human’ name, these AI and ML-driven bots can literally converse with humans
- Web crawlers or spiders that enhance website search results by scanning, indexing and extracting web content
- Scrapers that are used largely by e-commerce businesses to download relevant product information like prices as well as process sentiment analysis from social media platform feeds
- Shopping bots that facilitate consumers with recommendations on best deals on products
- Monitoring bots that scan systems for malware and unusual or unauthorized activity
- Transaction bots that securely process online payments and process personal user data during online activity
Fraught with danger
Over time, bad actors have used the technology to program them to malicious ends. While good ones run in the background, without creating compromising situations, bad bots on the other hand undermine and compromise web security. Routinely disrupting business operations, and performing unauthorized actions they obfuscate information on traffic sources, generate misleading analytical reports, and drive bandwidth usage of the victim up.
Some of the more infamous bots include:
- Download bots that are programmed to download software or applications automatically to create misleading product ratings
- Spambots that are tasked with collecting a large number of email addresses, creating fake accounts, and posting spam messages on social media platforms
- Ticketing bots that are used by booking agencies to purchase tickets at lower prices and sell them at higher prices
- DDoS bots that are programmed to overwhelm systems and render the website or application ineffective to a large number of users
- Fraud bots that automatically click on malicious links that drive online business revenue
- File-sharing bots that provide recommendations with links to malicious sites using commonly searched terms on the internet
- Social media bots that indulge in false social media platform activity like follows, likes, etc. with a view to disseminate spam content
- Botnets are a group of malicious bots that work cohesively for nefarious purposes. Botnets perform tasks that call for a high volume of computing power and memory.
How they operate
Bots comprise a set of algorithms that help drive them. They reside in the network and deploy internet services to perform their tasks around the clock. Studies say that a significantly high volume of all internet traffic is essentially bot traffic. Barracuda (2) estimates it at two-thirds of all internet traffic, with almost 67% being bad bot traffic. Cloudfare (4) puts the figure at 40%, with a high percentage of this being malicious ones.
Large-scale bot activity is known as bot traffic, and though the term is often interpreted negatively, it can also be a case of good bots being at work on the network. Hence, the purpose of the bot is moot. Heavy bot traffic is identified using web analysis tools. Typically high bot traffic is seen in cases of inordinately high pageviews or hits, higher than normal click rates, high bounce rates, high traffic from a particular zone, junk conversions due to the presence of spam bots, and spurts in activity or periods of unusual inactivity.
Keeping an eye out for them
Bots have the potential to create unhealthy and compromising situations affecting business continuity and operations. SOCs therefore keep a close eye on such symptoms as:
- Spike or drop in site traffic. Heavy traffic can indicate a DDoS attack, while a drop in traffic can be indicative of a scraper bot at work that can impact the website’s rankings
- Lower page views can be indicative of click fraud bots that disguise themselves as genuine users
- Zero engagement time on a site could be the handiwork of a bot intended to drive up hit rates
- Higher conversion rates caused by credential stuffing by bots populating online web forms
- Unknown referrals with no identifiable source could be the consequence of a bot attack
- Traffic from hitherto unknown locations may
Managing them
It is now almost a given that organizations and SOCs have to live with them, a digital necessary evil. Analytic tools like Google Analytics do make available an option to ‘exclude all hits from known bots and spiders’, but their increasing number invariably makes this an arduous task. Imperva (5) suggests the following methods to control bots:
- Use of captcha codes that prompt user input which cannot be populated by an automated
- Block hosting services and servers known to contain them
- Protect every bad bot access point
- Carefully evaluate traffic sources
- Investigate traffic spikes and set up alerts to notify in event of abnormalities
- Evaluate bot mitigation systems
End words
Forbes (6) calls them the ultimate hack, consistently slipping under the radar, and offering scammers the opportunity to monetize their nefarious intent. Bots as a service is making it easy for them to orchestrate their actions for a fee. Organizations are seized of their dangers but also are finding it hard to disengage, given the benefits they confer. The statistics speak for themselves. 96% (7) say good bots are important to the success of their organization’s eCommerce, yet 80% have lost revenue to bad ones. 81% of businesses often or very often deal with issues related to malicious ones.
Looking back to 2004 when Arsenal created English Premier League history by winning the 2004 edition, earning them the sobriquet The Invincibles, their investment in Denis Bergkamp was justified many times over. Though the talented player hardly scored during the competition, his vision, skills, and assists proved that the good and the bad can often go together.
Organizations with good risk appetite will concur.
References:
- What is a Bot? – Types of Bots Explained – AWS (amazon.com)
- What are bots and how do they work? (techtarget.com)
- How & Why To Prevent Bots From Crawling Your Site (searchenginejournal.com)
- What is bot traffic? | How to stop bot traffic | Cloudflare
- 9 Recommendations to Prevent Bad Bots on Your Website | Imperva
- Twitter’s Bots Were The Business All Along (forbes.com)
- The State Of Bot Cybersecurity, 2020 (forbes.com)
