Posted: Tue, Apr 17, 2018 at 10:35 AM - Showing 337 Views
So I have noticed, That my network was getting popular, More popular then its warrant or result. So I went to work building tools to help me understand where why and what is happening on my network of sites.
I came across something rather absurd.
Robots are doing more then just scanning your site for the content. They are doing searches to see what they can find. and no this isn't google. Hunting down the cause and alot of nameless robots that run off a open API just slam my network with garbage questions to my search query.
So I decided to take a look at this not as if it was a threat. But instead an opportunity. Why are these bots doing what they are doing? What possible thing could they be doing by throwing such trash search questions into the system.
Ive learned these bots also dont parse the information they ask for. after calculated testing and temp clock bans to change to result to a denial of service the bots would continue to blindly request the information.
In one instance for 1 hour a robot was presented with a message "You are banned from the network" Was the only output given to the robot for a present of 1 hour. In that hour the robot continued requests to the search box with the same result of being banned. The bot attempted over 4000 queries un-bothered by its results.
Shortly after I tried to feed the bots with controlled redirect options. Where the bot would in theory search the word "Pie" and the result would remain
1. Result 1 - URL 1
2. Result 2 - URL 2
3. Result 3 - URL 3
And during that time the robots did not even navigate to the control group.
These are my findings so far. But the bots appear to just be trash bots right now. Who would make them and why do they exist is something im still researching.